task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-6x9s5-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-6x9s5-kubeconfig [get-kubeconfig] Found admin password secret: cluster-6x9s5-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-6x9s5-password [get-kubeconfig] API Server URL: https://a9e5214c7506644cb80ef560edbc437e-108a202261685530.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.708a374e-9898-4f03-b4f0-15036756e60c.prod.konfluxeaas.com [clone-repo] install-llm-d-dashboards [clone-repo] https://github.com/tessapham/kserve [clone-repo] Cloning into '/workspace/source'... [clone-repo] Updating files: 62% (1950/3100) Updating files: 63% (1953/3100) Updating files: 64% (1984/3100) Updating files: 65% (2015/3100) Updating files: 66% (2046/3100) Updating files: 67% (2077/3100) Updating files: 68% (2108/3100) Updating files: 69% (2139/3100) Updating files: 70% (2170/3100) Updating files: 71% (2201/3100) Updating files: 72% (2232/3100) Updating files: 73% (2263/3100) Updating files: 74% (2294/3100) Updating files: 75% (2325/3100) Updating files: 76% (2356/3100) Updating files: 77% (2387/3100) Updating files: 78% (2418/3100) Updating files: 79% (2449/3100) Updating files: 80% (2480/3100) Updating files: 81% (2511/3100) Updating files: 82% (2542/3100) Updating files: 83% (2573/3100) Updating files: 84% (2604/3100) Updating files: 85% (2635/3100) Updating files: 86% (2666/3100) Updating files: 87% (2697/3100) Updating files: 88% (2728/3100) Updating files: 89% (2759/3100) Updating files: 90% (2790/3100) Updating files: 91% (2821/3100) Updating files: 92% (2852/3100) Updating files: 93% (2883/3100) Updating files: 94% (2914/3100) Updating files: 95% (2945/3100) Updating files: 96% (2976/3100) Updating files: 97% (3007/3100) Updating files: 98% (3038/3100) Updating files: 99% (3069/3100) Updating files: 100% (3100/3100) Updating files: 100% (3100/3100), done. [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:0706c86054be2f22ee66c8739e97daebd0094b98a978f91cd3fb4f59862bad3f [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:0706c86054be2f22ee66c8739e97daebd0094b98a978f91cd3fb4f59862bad3f [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:94a52ece937ab7ef2143573dbb2c4bc4ee13e762dc2ec9488b14edc9f48edab9 [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:94a52ece937ab7ef2143573dbb2c4bc4ee13e762dc2ec9488b14edc9f48edab9 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:a8d64b572a976cc85b28488a3ef8f7d50ce28ddb75186c1859612e88d6dba080 [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:a8d64b572a976cc85b28488a3ef8f7d50ce28ddb75186c1859612e88d6dba080 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:0dc9b5551b05cf2174c007411651a089c1777638fec85e0a699de0f096e2706a [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:0dc9b5551b05cf2174c007411651a089c1777638fec85e0a699de0f096e2706a [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu and not autoscaling and not tracing' 2 llm-d [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1703 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1703 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1703 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 266 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading cryptography (4.3MiB) [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Prepared 101 packages in 1.94s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 101 packages in 369ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.13.3 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.1.8 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==46.0.5 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.19.0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.10.16 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn1==0.6.3 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.12.1 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.22 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==1.2.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.6.2 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 92ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Now using project "kserve" on server "https://a9e5214c7506644cb80ef560edbc437e-108a202261685530.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (25/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (50/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (55/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-68bff59c-jl7h5 1/1 Running 0 41s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-68bff59c-jl7h5 condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-559f75f947-66hkd 1/1 Running 0 45s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-559f75f947-66hkd condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-5fbd5c4644-47c8s 1/1 Running 0 51s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-5fbd5c4644-47c8s condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:94a52ece937ab7ef2143573dbb2c4bc4ee13e762dc2ec9488b14edc9f48edab9 [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:0dc9b5551b05cf2174c007411651a089c1777638fec85e0a699de0f096e2706a [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:0706c86054be2f22ee66c8739e97daebd0094b98a978f91cd3fb4f59862bad3f [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:a8d64b572a976cc85b28488a3ef8f7d50ce28ddb75186c1859612e88d6dba080 [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-7f65c45d5c-7t766 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-7f65c45d5c-7t766 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-applying LLMInferenceServiceConfig resources with webhook validation... [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-scheduler is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.23) [e2e-llm-inference-service] 🎯 Server version (4.21.23) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (15/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (20/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (25/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (30/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (35/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-r25kc 1/1 Running 0 11s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-r25kc condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-94bb8fbfd-zmx9k 1/1 Running 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-94bb8fbfd-zmx9k condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-9455cf8d5-vmnww 1/1 Running 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-9455cf8d5-vmnww condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d85f6564-gmblh 1/1 Running 0 79s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d85f6564-gmblh condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d85f6564-gmblh 1/1 Running 0 89s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d85f6564-gmblh condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-68bc99fcb8-2k89z" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-68bc99fcb8-7bghc condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.23 [e2e-llm-inference-service] Kubernetes Version: v1.34.8 [e2e-llm-inference-service] ClusterVersion desired: 4.21.23 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.23 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-1 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.1 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.0 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.0 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.1 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:94a52ece937ab7ef2143573dbb2c4bc4ee13e762dc2ec9488b14edc9f48edab9 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:03e30413b0c45e6a0fe1d4c4ba6c8528a568934fd093517976e004dcc082e055 [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:fast [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:1f4d2febe8e72e717f8b828958e404b656525a5cec24af58abe2a92f3e9d7e19 [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:0dc9b5551b05cf2174c007411651a089c1777638fec85e0a699de0f096e2706a [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:0dc9b5551b05cf2174c007411651a089c1777638fec85e0a699de0f096e2706a [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu and not autoscaling and not tracing [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.122.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'json-report': '1.5.0', 'metadata': '3.1.1', 'httpx': '0.30.0', 'cov': '5.0.0', 'asyncio': '0.23.8', 'xdist': '3.6.1', 'anyio': '4.9.0'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: json-report-1.5.0, metadata-3.1.1, httpx-0.30.0, cov-5.0.0, asyncio-0.23.8, xdist-3.6.1, anyio-4.9.0 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [42 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] 2026-07-06 20:47:11.284 6669 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-06 20:47:11.284 6672 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-06 20:47:11.297 6669 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 20:47:11.297 6669 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 20:47:11.321 6669 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-06 20:47:11.321 6669 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-06 20:47:11.329 6669 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-06 20:47:11.329 6669 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-07-06 20:47:46.923 6669 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 20:47:46.923 6669 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 20:47:46.968 6669 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-06 20:47:46.968 6669 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] 2026-07-06 20:57:24.517 6669 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 20:57:24.517 6669 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 20:57:24.563 6669 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-06 20:57:24.563 6669 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] 2026-07-06 21:12:09.223 6669 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:12:09.223 6669 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:12:09.264 6669 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-06 21:12:09.264 6669 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-06 21:12:09.264 6669 kserve.trace Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:12:09.264 6669 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:12:09.267 6669 kserve.trace Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-06 21:12:09.267 6669 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-06 21:12:09.276 6669 kserve.trace ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-06 21:12:09.276 6669 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-06 21:12:09.277 6669 kserve.trace Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:12:09.277 6669 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:12:09.282 6669 kserve.trace Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-06 21:12:09.282 6669 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-06 21:12:09.291 6669 kserve.trace ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-06 21:12:09.291 6669 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] 2026-07-06 21:41:25.749 6669 kserve.trace Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:41:25.749 6669 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:41:25.782 6669 kserve.trace Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-06 21:41:25.782 6669 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-06 21:41:25.788 6669 kserve.trace ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-06 21:41:25.788 6669 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-06 21:41:25.789 6669 kserve.trace Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:41:25.789 6669 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:41:25.791 6669 kserve.trace Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-06 21:41:25.791 6669 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-06 21:41:25.804 6669 kserve.trace ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-06 21:41:25.804 6669 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-06 21:41:25.804 6669 kserve.trace Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:41:25.804 6669 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:41:25.807 6669 kserve.trace Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-06 21:41:25.807 6669 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-06 21:41:25.817 6669 kserve.trace ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-06 21:41:25.817 6669 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] _____________ test_llm_with_lora_adapters[multiple-lora-adapters] ______________ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] conn = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "Explain machine learning in simple terms.", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '104', 'Content-Type': 'application/json'} [e2e-llm-inference-service] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), chunked = False [e2e-llm-inference-service] response_conn = [e2e-llm-inference-service] preload_content = False, decode_content = False, enforce_content_length = True [e2e-llm-inference-service] [e2e-llm-inference-service] def _make_request( [e2e-llm-inference-service] self, [e2e-llm-inference-service] conn: BaseHTTPConnection, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | None = None, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] response_conn: BaseHTTPConnection | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] enforce_content_length: bool = True, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Perform a request on a given urllib connection object taken from our [e2e-llm-inference-service] pool. [e2e-llm-inference-service] [e2e-llm-inference-service] :param conn: [e2e-llm-inference-service] a connection from one of our connection pools [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] Pass ``None`` to retry until you receive a response. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param response_conn: [e2e-llm-inference-service] Set this to ``None`` if you will handle releasing the connection or [e2e-llm-inference-service] set the connection to have the response release it. [e2e-llm-inference-service] [e2e-llm-inference-service] :param preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded during construction. [e2e-llm-inference-service] [e2e-llm-inference-service] :param decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param enforce_content_length: [e2e-llm-inference-service] Enforce content length checking. Body returned by server must match [e2e-llm-inference-service] value of Content-Length header, if present. Otherwise, raise error. [e2e-llm-inference-service] """ [e2e-llm-inference-service] self.num_requests += 1 [e2e-llm-inference-service] [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] timeout_obj.start_connect() [e2e-llm-inference-service] conn.timeout = Timeout.resolve_default_timeout(timeout_obj.connect_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Trigger any extra validation we need to do. [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._validate_conn(conn) [e2e-llm-inference-service] except (SocketTimeout, BaseSSLError) as e: [e2e-llm-inference-service] self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # _validate_conn() starts the connection to an HTTPS proxy [e2e-llm-inference-service] # so we need to wrap errors with 'ProxyError' here too. [e2e-llm-inference-service] except ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] # If the connection didn't successfully connect to it's proxy [e2e-llm-inference-service] # then there [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, (OSError, NewConnectionError, TimeoutError, SSLError) [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] raise new_e [e2e-llm-inference-service] [e2e-llm-inference-service] # conn.request() calls http.client.*.request, not the method in [e2e-llm-inference-service] # urllib3.request. It also calls makefile (recv) on the socket. [e2e-llm-inference-service] try: [e2e-llm-inference-service] conn.request( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] enforce_content_length=enforce_content_length, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # We are swallowing BrokenPipeError (errno.EPIPE) since the server is [e2e-llm-inference-service] # legitimately able to close the connection after sending a valid response. [e2e-llm-inference-service] # With this behaviour, the received response is still readable. [e2e-llm-inference-service] except BrokenPipeError: [e2e-llm-inference-service] pass [e2e-llm-inference-service] except OSError as e: [e2e-llm-inference-service] # MacOS/Linux [e2e-llm-inference-service] # EPROTOTYPE and ECONNRESET are needed on macOS [e2e-llm-inference-service] # https://erickt.github.io/blog/2014/11/19/adventures-in-debugging-a-potential-osx-kernel-bug/ [e2e-llm-inference-service] # Condition changed later to emit ECONNRESET instead of only EPROTOTYPE. [e2e-llm-inference-service] if e.errno != errno.EPROTOTYPE and e.errno != errno.ECONNRESET: [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # Reset the timeout for the recv() on the socket [e2e-llm-inference-service] read_timeout = timeout_obj.read_timeout [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn.is_closed: [e2e-llm-inference-service] # In Python 3 socket.py will catch EAGAIN and return None when you [e2e-llm-inference-service] # try and read into the file pointer created by http.client, which [e2e-llm-inference-service] # instead raises a BadStatusLine exception. Instead of catching [e2e-llm-inference-service] # the exception and assuming all BadStatusLine exceptions are read [e2e-llm-inference-service] # timeouts, check for a zero timeout before making the request. [e2e-llm-inference-service] if read_timeout == 0: [e2e-llm-inference-service] raise ReadTimeoutError( [e2e-llm-inference-service] self, url, f"Read timed out. (read timeout={read_timeout})" [e2e-llm-inference-service] ) [e2e-llm-inference-service] conn.timeout = read_timeout [e2e-llm-inference-service] [e2e-llm-inference-service] # Receive the response from the server [e2e-llm-inference-service] try: [e2e-llm-inference-service] > response = conn.getresponse() [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:534: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] [e2e-llm-inference-service] def getresponse( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] ) -> HTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get the response from the server. [e2e-llm-inference-service] [e2e-llm-inference-service] If the HTTPConnection is in the correct state, returns an instance of HTTPResponse or of whatever object is returned by the response_class variable. [e2e-llm-inference-service] [e2e-llm-inference-service] If a request has not been sent or if a previous response has not be handled, ResponseNotReady is raised. If the HTTP response indicates that the connection should be closed, then it will be closed before the response is returned. When the connection is closed, the underlying socket is closed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] # Raise the same error as http.client.HTTPConnection [e2e-llm-inference-service] if self._response_options is None: [e2e-llm-inference-service] raise ResponseNotReady() [e2e-llm-inference-service] [e2e-llm-inference-service] # Reset this attribute for being used again. [e2e-llm-inference-service] resp_options = self._response_options [e2e-llm-inference-service] self._response_options = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Since the connection's timeout value may have been updated [e2e-llm-inference-service] # we need to set the timeout on the socket. [e2e-llm-inference-service] self.sock.settimeout(self.timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] # This is needed here to avoid circular import errors [e2e-llm-inference-service] from .response import HTTPResponse [e2e-llm-inference-service] [e2e-llm-inference-service] # Save a reference to the shutdown function before ownership is passed [e2e-llm-inference-service] # to httplib_response [e2e-llm-inference-service] # TODO should we implement it everywhere? [e2e-llm-inference-service] _shutdown = getattr(self.sock, "shutdown", None) [e2e-llm-inference-service] [e2e-llm-inference-service] # Get the response from http.client.HTTPConnection [e2e-llm-inference-service] > httplib_response = super().getresponse() [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connection.py:571: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] [e2e-llm-inference-service] def getresponse(self): [e2e-llm-inference-service] """Get the response from the server. [e2e-llm-inference-service] [e2e-llm-inference-service] If the HTTPConnection is in the correct state, returns an [e2e-llm-inference-service] instance of HTTPResponse or of whatever object is returned by [e2e-llm-inference-service] the response_class variable. [e2e-llm-inference-service] [e2e-llm-inference-service] If a request has not been sent or if a previous response has [e2e-llm-inference-service] not be handled, ResponseNotReady is raised. If the HTTP [e2e-llm-inference-service] response indicates that the connection should be closed, then [e2e-llm-inference-service] it will be closed before the response is returned. When the [e2e-llm-inference-service] connection is closed, the underlying socket is closed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] # if a prior response has been completed, then forget about it. [e2e-llm-inference-service] if self.__response and self.__response.isclosed(): [e2e-llm-inference-service] self.__response = None [e2e-llm-inference-service] [e2e-llm-inference-service] # if a prior response exists, then it must be completed (otherwise, we [e2e-llm-inference-service] # cannot read this response's header to determine the connection-close [e2e-llm-inference-service] # behavior) [e2e-llm-inference-service] # [e2e-llm-inference-service] # note: if a prior response existed, but was connection-close, then the [e2e-llm-inference-service] # socket and response were made independent of this HTTPConnection [e2e-llm-inference-service] # object since a new request requires that we open a whole new [e2e-llm-inference-service] # connection [e2e-llm-inference-service] # [e2e-llm-inference-service] # this means the prior response had one of two states: [e2e-llm-inference-service] # 1) will_close: this connection was reset and the prior socket and [e2e-llm-inference-service] # response operate independently [e2e-llm-inference-service] # 2) persistent: the response was retained and we await its [e2e-llm-inference-service] # isclosed() status to become true. [e2e-llm-inference-service] # [e2e-llm-inference-service] if self.__state != _CS_REQ_SENT or self.__response: [e2e-llm-inference-service] raise ResponseNotReady(self.__state) [e2e-llm-inference-service] [e2e-llm-inference-service] if self.debuglevel > 0: [e2e-llm-inference-service] response = self.response_class(self.sock, self.debuglevel, [e2e-llm-inference-service] method=self._method) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = self.response_class(self.sock, method=self._method) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > response.begin() [e2e-llm-inference-service] [e2e-llm-inference-service] /usr/lib64/python3.11/http/client.py:1395: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] [e2e-llm-inference-service] def begin(self): [e2e-llm-inference-service] if self.headers is not None: [e2e-llm-inference-service] # we've already started reading the response [e2e-llm-inference-service] return [e2e-llm-inference-service] [e2e-llm-inference-service] # read until we get a non-100 response [e2e-llm-inference-service] while True: [e2e-llm-inference-service] > version, status, reason = self._read_status() [e2e-llm-inference-service] [e2e-llm-inference-service] /usr/lib64/python3.11/http/client.py:325: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] [e2e-llm-inference-service] def _read_status(self): [e2e-llm-inference-service] > line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") [e2e-llm-inference-service] [e2e-llm-inference-service] /usr/lib64/python3.11/http/client.py:286: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] b = [e2e-llm-inference-service] [e2e-llm-inference-service] def readinto(self, b): [e2e-llm-inference-service] """Read up to len(b) bytes into the writable buffer *b* and return [e2e-llm-inference-service] the number of bytes read. If the socket is non-blocking and no bytes [e2e-llm-inference-service] are available, None is returned. [e2e-llm-inference-service] [e2e-llm-inference-service] If *b* is non-empty, a 0 return value indicates that the connection [e2e-llm-inference-service] was shutdown at the other end. [e2e-llm-inference-service] """ [e2e-llm-inference-service] self._checkClosed() [e2e-llm-inference-service] self._checkReadable() [e2e-llm-inference-service] if self._timeout_occurred: [e2e-llm-inference-service] raise OSError("cannot read from timed out object") [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return self._sock.recv_into(b) [e2e-llm-inference-service] E TimeoutError: timed out [e2e-llm-inference-service] [e2e-llm-inference-service] /usr/lib64/python3.11/socket.py:718: TimeoutError [e2e-llm-inference-service] [e2e-llm-inference-service] The above exception was the direct cause of the following exception: [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "Explain machine learning in simple terms.", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '104', 'Content-Type': 'application/json'} [e2e-llm-inference-service] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False, err = None, clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] > response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:787: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] conn = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "Explain machine learning in simple terms.", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '104', 'Content-Type': 'application/json'} [e2e-llm-inference-service] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), chunked = False [e2e-llm-inference-service] response_conn = [e2e-llm-inference-service] preload_content = False, decode_content = False, enforce_content_length = True [e2e-llm-inference-service] [e2e-llm-inference-service] def _make_request( [e2e-llm-inference-service] self, [e2e-llm-inference-service] conn: BaseHTTPConnection, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | None = None, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] response_conn: BaseHTTPConnection | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] enforce_content_length: bool = True, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Perform a request on a given urllib connection object taken from our [e2e-llm-inference-service] pool. [e2e-llm-inference-service] [e2e-llm-inference-service] :param conn: [e2e-llm-inference-service] a connection from one of our connection pools [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] Pass ``None`` to retry until you receive a response. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param response_conn: [e2e-llm-inference-service] Set this to ``None`` if you will handle releasing the connection or [e2e-llm-inference-service] set the connection to have the response release it. [e2e-llm-inference-service] [e2e-llm-inference-service] :param preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded during construction. [e2e-llm-inference-service] [e2e-llm-inference-service] :param decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param enforce_content_length: [e2e-llm-inference-service] Enforce content length checking. Body returned by server must match [e2e-llm-inference-service] value of Content-Length header, if present. Otherwise, raise error. [e2e-llm-inference-service] """ [e2e-llm-inference-service] self.num_requests += 1 [e2e-llm-inference-service] [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] timeout_obj.start_connect() [e2e-llm-inference-service] conn.timeout = Timeout.resolve_default_timeout(timeout_obj.connect_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Trigger any extra validation we need to do. [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._validate_conn(conn) [e2e-llm-inference-service] except (SocketTimeout, BaseSSLError) as e: [e2e-llm-inference-service] self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # _validate_conn() starts the connection to an HTTPS proxy [e2e-llm-inference-service] # so we need to wrap errors with 'ProxyError' here too. [e2e-llm-inference-service] except ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] # If the connection didn't successfully connect to it's proxy [e2e-llm-inference-service] # then there [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, (OSError, NewConnectionError, TimeoutError, SSLError) [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] raise new_e [e2e-llm-inference-service] [e2e-llm-inference-service] # conn.request() calls http.client.*.request, not the method in [e2e-llm-inference-service] # urllib3.request. It also calls makefile (recv) on the socket. [e2e-llm-inference-service] try: [e2e-llm-inference-service] conn.request( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] enforce_content_length=enforce_content_length, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # We are swallowing BrokenPipeError (errno.EPIPE) since the server is [e2e-llm-inference-service] # legitimately able to close the connection after sending a valid response. [e2e-llm-inference-service] # With this behaviour, the received response is still readable. [e2e-llm-inference-service] except BrokenPipeError: [e2e-llm-inference-service] pass [e2e-llm-inference-service] except OSError as e: [e2e-llm-inference-service] # MacOS/Linux [e2e-llm-inference-service] # EPROTOTYPE and ECONNRESET are needed on macOS [e2e-llm-inference-service] # https://erickt.github.io/blog/2014/11/19/adventures-in-debugging-a-potential-osx-kernel-bug/ [e2e-llm-inference-service] # Condition changed later to emit ECONNRESET instead of only EPROTOTYPE. [e2e-llm-inference-service] if e.errno != errno.EPROTOTYPE and e.errno != errno.ECONNRESET: [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # Reset the timeout for the recv() on the socket [e2e-llm-inference-service] read_timeout = timeout_obj.read_timeout [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn.is_closed: [e2e-llm-inference-service] # In Python 3 socket.py will catch EAGAIN and return None when you [e2e-llm-inference-service] # try and read into the file pointer created by http.client, which [e2e-llm-inference-service] # instead raises a BadStatusLine exception. Instead of catching [e2e-llm-inference-service] # the exception and assuming all BadStatusLine exceptions are read [e2e-llm-inference-service] # timeouts, check for a zero timeout before making the request. [e2e-llm-inference-service] if read_timeout == 0: [e2e-llm-inference-service] raise ReadTimeoutError( [e2e-llm-inference-service] self, url, f"Read timed out. (read timeout={read_timeout})" [e2e-llm-inference-service] ) [e2e-llm-inference-service] conn.timeout = read_timeout [e2e-llm-inference-service] [e2e-llm-inference-service] # Receive the response from the server [e2e-llm-inference-service] try: [e2e-llm-inference-service] response = conn.getresponse() [e2e-llm-inference-service] except (BaseSSLError, OSError) as e: [e2e-llm-inference-service] > self._raise_timeout(err=e, url=url, timeout_value=read_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:536: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] err = TimeoutError('timed out') [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions' [e2e-llm-inference-service] timeout_value = 60 [e2e-llm-inference-service] [e2e-llm-inference-service] def _raise_timeout( [e2e-llm-inference-service] self, [e2e-llm-inference-service] err: BaseSSLError | OSError | SocketTimeout, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] timeout_value: _TYPE_TIMEOUT | None, [e2e-llm-inference-service] ) -> None: [e2e-llm-inference-service] """Is the error actually a timeout? Will raise a ReadTimeout or pass""" [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(err, SocketTimeout): [e2e-llm-inference-service] > raise ReadTimeoutError( [e2e-llm-inference-service] self, url, f"Read timed out. (read timeout={timeout_value})" [e2e-llm-inference-service] ) from err [e2e-llm-inference-service] E urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:367: ReadTimeoutError [e2e-llm-inference-service] [e2e-llm-inference-service] The above exception was the direct cause of the following exception: [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] request = , stream = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), verify = '/tmp/ca.crt' [e2e-llm-inference-service] cert = None, proxies = OrderedDict() [e2e-llm-inference-service] [e2e-llm-inference-service] def send( [e2e-llm-inference-service] self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None [e2e-llm-inference-service] ): [e2e-llm-inference-service] """Sends PreparedRequest object. Returns Response object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param request: The :class:`PreparedRequest ` being sent. [e2e-llm-inference-service] :param stream: (optional) Whether to stream the request content. [e2e-llm-inference-service] :param timeout: (optional) How long to wait for the server to send [e2e-llm-inference-service] data before giving up, as a float, or a :ref:`(connect timeout, [e2e-llm-inference-service] read timeout) ` tuple. [e2e-llm-inference-service] :type timeout: float or tuple or urllib3 Timeout object [e2e-llm-inference-service] :param verify: (optional) Either a boolean, in which case it controls whether [e2e-llm-inference-service] we verify the server's TLS certificate, or a string, in which case it [e2e-llm-inference-service] must be a path to a CA bundle to use [e2e-llm-inference-service] :param cert: (optional) Any user-provided SSL certificate to be trusted. [e2e-llm-inference-service] :param proxies: (optional) The proxies dictionary to apply to the request. [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] conn = self.get_connection_with_tls_context( [e2e-llm-inference-service] request, verify, proxies=proxies, cert=cert [e2e-llm-inference-service] ) [e2e-llm-inference-service] except LocationValueError as e: [e2e-llm-inference-service] raise InvalidURL(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] self.cert_verify(conn, request.url, verify, cert) [e2e-llm-inference-service] url = self.request_url(request, proxies) [e2e-llm-inference-service] self.add_headers( [e2e-llm-inference-service] request, [e2e-llm-inference-service] stream=stream, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] verify=verify, [e2e-llm-inference-service] cert=cert, [e2e-llm-inference-service] proxies=proxies, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] chunked = not (request.body is None or "Content-Length" in request.headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(timeout, tuple): [e2e-llm-inference-service] try: [e2e-llm-inference-service] connect, read = timeout [e2e-llm-inference-service] timeout = TimeoutSauce(connect=connect, read=read) [e2e-llm-inference-service] except ValueError: [e2e-llm-inference-service] raise ValueError( [e2e-llm-inference-service] f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, " [e2e-llm-inference-service] f"or a single float to set both timeouts to the same value." [e2e-llm-inference-service] ) [e2e-llm-inference-service] elif isinstance(timeout, TimeoutSauce): [e2e-llm-inference-service] pass [e2e-llm-inference-service] else: [e2e-llm-inference-service] timeout = TimeoutSauce(connect=timeout, read=timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > resp = conn.urlopen( [e2e-llm-inference-service] method=request.method, [e2e-llm-inference-service] url=url, [e2e-llm-inference-service] body=request.body, [e2e-llm-inference-service] headers=request.headers, [e2e-llm-inference-service] redirect=False, [e2e-llm-inference-service] assert_same_host=False, [e2e-llm-inference-service] preload_content=False, [e2e-llm-inference-service] decode_content=False, [e2e-llm-inference-service] retries=self.max_retries, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/adapters.py:667: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "Explain machine learning in simple terms.", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '104', 'Content-Type': 'application/json'} [e2e-llm-inference-service] retries = Retry(total=7, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "Explain machine learning in simple terms.", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '104', 'Content-Type': 'application/json'} [e2e-llm-inference-service] retries = Retry(total=6, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "Explain machine learning in simple terms.", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '104', 'Content-Type': 'application/json'} [e2e-llm-inference-service] retries = Retry(total=5, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "Explain machine learning in simple terms.", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '104', 'Content-Type': 'application/json'} [e2e-llm-inference-service] retries = Retry(total=4, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "Explain machine learning in simple terms.", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '104', 'Content-Type': 'application/json'} [e2e-llm-inference-service] retries = Retry(total=3, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "Explain machine learning in simple terms.", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '104', 'Content-Type': 'application/json'} [e2e-llm-inference-service] retries = Retry(total=2, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "Explain machine learning in simple terms.", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '104', 'Content-Type': 'application/json'} [e2e-llm-inference-service] retries = Retry(total=1, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "Explain machine learning in simple terms.", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '104', 'Content-Type': 'application/json'} [e2e-llm-inference-service] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False [e2e-llm-inference-service] err = ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)") [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] retries.sleep() [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of the error for the retry warning. [e2e-llm-inference-service] err = e [e2e-llm-inference-service] [e2e-llm-inference-service] finally: [e2e-llm-inference-service] if not clean_exit: [e2e-llm-inference-service] # We hit some kind of exception, handled or otherwise. We need [e2e-llm-inference-service] # to throw the connection away unless explicitly told not to. [e2e-llm-inference-service] # Close the connection, set the variable to None, and make sure [e2e-llm-inference-service] # we put the None back in the pool to avoid leaking it. [e2e-llm-inference-service] if conn: [e2e-llm-inference-service] conn.close() [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] release_this_conn = True [e2e-llm-inference-service] [e2e-llm-inference-service] if release_this_conn: [e2e-llm-inference-service] # Put the connection back to be reused. If the connection is [e2e-llm-inference-service] # expired then it will be None, which will get replaced with a [e2e-llm-inference-service] # fresh connection during _get_conn. [e2e-llm-inference-service] self._put_conn(conn) [e2e-llm-inference-service] [e2e-llm-inference-service] if not conn: [e2e-llm-inference-service] # Try again [e2e-llm-inference-service] log.warning( [e2e-llm-inference-service] "Retrying (%r) after connection broken by '%r': %s", retries, err, url [e2e-llm-inference-service] ) [e2e-llm-inference-service] > return self.urlopen( [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] body, [e2e-llm-inference-service] headers, [e2e-llm-inference-service] retries, [e2e-llm-inference-service] redirect, [e2e-llm-inference-service] assert_same_host, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] pool_timeout=pool_timeout, [e2e-llm-inference-service] release_conn=release_conn, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] body_pos=body_pos, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:871: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions' [e2e-llm-inference-service] body = b'{"model": "facebook/opt-125m", "prompt": "Explain machine learning in simple terms.", "max_tokens": 100}' [e2e-llm-inference-service] headers = {'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '104', 'Content-Type': 'application/json'} [e2e-llm-inference-service] retries = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] redirect = False, assert_same_host = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None [e2e-llm-inference-service] release_conn = False, chunked = False, body_pos = None, preload_content = False [e2e-llm-inference-service] decode_content = False, response_kw = {} [e2e-llm-inference-service] parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions', query=None, fragment=None) [e2e-llm-inference-service] destination_scheme = None, conn = None, release_this_conn = True [e2e-llm-inference-service] http_tunnel_required = False, err = None, clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] def urlopen( # type: ignore[override] [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str, [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] body: _TYPE_BODY | None = None, [e2e-llm-inference-service] headers: typing.Mapping[str, str] | None = None, [e2e-llm-inference-service] retries: Retry | bool | int | None = None, [e2e-llm-inference-service] redirect: bool = True, [e2e-llm-inference-service] assert_same_host: bool = True, [e2e-llm-inference-service] timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT, [e2e-llm-inference-service] pool_timeout: int | None = None, [e2e-llm-inference-service] release_conn: bool | None = None, [e2e-llm-inference-service] chunked: bool = False, [e2e-llm-inference-service] body_pos: _TYPE_BODY_POSITION | None = None, [e2e-llm-inference-service] preload_content: bool = True, [e2e-llm-inference-service] decode_content: bool = True, [e2e-llm-inference-service] **response_kw: typing.Any, [e2e-llm-inference-service] ) -> BaseHTTPResponse: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Get a connection from the pool and perform an HTTP request. This is the [e2e-llm-inference-service] lowest level call for making a request, so you'll need to specify all [e2e-llm-inference-service] the raw details. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] More commonly, it's appropriate to use a convenience method [e2e-llm-inference-service] such as :meth:`request`. [e2e-llm-inference-service] [e2e-llm-inference-service] .. note:: [e2e-llm-inference-service] [e2e-llm-inference-service] `release_conn` will only behave as expected if [e2e-llm-inference-service] `preload_content=False` because we want to make [e2e-llm-inference-service] `preload_content=False` the default behaviour someday soon without [e2e-llm-inference-service] breaking backwards compatibility. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: [e2e-llm-inference-service] HTTP request method (such as GET, POST, PUT, etc.) [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: [e2e-llm-inference-service] The URL to perform the request on. [e2e-llm-inference-service] [e2e-llm-inference-service] :param body: [e2e-llm-inference-service] Data to send in the request body, either :class:`str`, :class:`bytes`, [e2e-llm-inference-service] an iterable of :class:`str`/:class:`bytes`, or a file-like object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param headers: [e2e-llm-inference-service] Dictionary of custom headers to send, such as User-Agent, [e2e-llm-inference-service] If-None-Match, etc. If None, pool headers are used. If provided, [e2e-llm-inference-service] these headers completely replace any pool-specific headers. [e2e-llm-inference-service] [e2e-llm-inference-service] :param retries: [e2e-llm-inference-service] Configure the number of retries to allow before raising a [e2e-llm-inference-service] :class:`~urllib3.exceptions.MaxRetryError` exception. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a [e2e-llm-inference-service] :class:`~urllib3.util.retry.Retry` object for fine-grained control [e2e-llm-inference-service] over different types of retries. [e2e-llm-inference-service] Pass an integer number to retry connection errors that many times, [e2e-llm-inference-service] but no other types of errors. Pass zero to never retry. [e2e-llm-inference-service] [e2e-llm-inference-service] If ``False``, then retries are disabled and any exception is raised [e2e-llm-inference-service] immediately. Also, instead of raising a MaxRetryError on redirects, [e2e-llm-inference-service] the redirect response will be returned. [e2e-llm-inference-service] [e2e-llm-inference-service] :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int. [e2e-llm-inference-service] [e2e-llm-inference-service] :param redirect: [e2e-llm-inference-service] If True, automatically handle redirects (status codes 301, 302, [e2e-llm-inference-service] 303, 307, 308). Each redirect counts as a retry. Disabling retries [e2e-llm-inference-service] will disable redirect, too. [e2e-llm-inference-service] [e2e-llm-inference-service] :param assert_same_host: [e2e-llm-inference-service] If ``True``, will make sure that the host of the pool requests is [e2e-llm-inference-service] consistent else will raise HostChangedError. When ``False``, you can [e2e-llm-inference-service] use the pool on an HTTP proxy and request foreign hosts. [e2e-llm-inference-service] [e2e-llm-inference-service] :param timeout: [e2e-llm-inference-service] If specified, overrides the default timeout for this one [e2e-llm-inference-service] request. It may be a float (in seconds) or an instance of [e2e-llm-inference-service] :class:`urllib3.util.Timeout`. [e2e-llm-inference-service] [e2e-llm-inference-service] :param pool_timeout: [e2e-llm-inference-service] If set and the pool is set to block=True, then this method will [e2e-llm-inference-service] block for ``pool_timeout`` seconds and raise EmptyPoolError if no [e2e-llm-inference-service] connection is available within the time period. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool preload_content: [e2e-llm-inference-service] If True, the response's body will be preloaded into memory. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool decode_content: [e2e-llm-inference-service] If True, will attempt to decode the body based on the [e2e-llm-inference-service] 'content-encoding' header. [e2e-llm-inference-service] [e2e-llm-inference-service] :param release_conn: [e2e-llm-inference-service] If False, then the urlopen call will not release the connection [e2e-llm-inference-service] back into the pool once a response is received (but will release if [e2e-llm-inference-service] you read the entire contents of the response such as when [e2e-llm-inference-service] `preload_content=True`). This is useful if you're not preloading [e2e-llm-inference-service] the response's content immediately. You will need to call [e2e-llm-inference-service] ``r.release_conn()`` on the response ``r`` to return the connection [e2e-llm-inference-service] back into the pool. If None, it takes the value of ``preload_content`` [e2e-llm-inference-service] which defaults to ``True``. [e2e-llm-inference-service] [e2e-llm-inference-service] :param bool chunked: [e2e-llm-inference-service] If True, urllib3 will send the body using chunked transfer [e2e-llm-inference-service] encoding. Otherwise, urllib3 will send the body using the standard [e2e-llm-inference-service] content-length form. Defaults to False. [e2e-llm-inference-service] [e2e-llm-inference-service] :param int body_pos: [e2e-llm-inference-service] Position to seek to in file-like body in the event of a retry or [e2e-llm-inference-service] redirect. Typically this won't need to be set because urllib3 will [e2e-llm-inference-service] auto-populate the value when needed. [e2e-llm-inference-service] """ [e2e-llm-inference-service] parsed_url = parse_url(url) [e2e-llm-inference-service] destination_scheme = parsed_url.scheme [e2e-llm-inference-service] [e2e-llm-inference-service] if headers is None: [e2e-llm-inference-service] headers = self.headers [e2e-llm-inference-service] [e2e-llm-inference-service] if not isinstance(retries, Retry): [e2e-llm-inference-service] retries = Retry.from_int(retries, redirect=redirect, default=self.retries) [e2e-llm-inference-service] [e2e-llm-inference-service] if release_conn is None: [e2e-llm-inference-service] release_conn = preload_content [e2e-llm-inference-service] [e2e-llm-inference-service] # Check host [e2e-llm-inference-service] if assert_same_host and not self.is_same_host(url): [e2e-llm-inference-service] raise HostChangedError(self, url, retries) [e2e-llm-inference-service] [e2e-llm-inference-service] # Ensure that the URL we're connecting to is properly encoded [e2e-llm-inference-service] if url.startswith("/"): [e2e-llm-inference-service] url = to_str(_encode_target(url)) [e2e-llm-inference-service] else: [e2e-llm-inference-service] url = to_str(parsed_url.url) [e2e-llm-inference-service] [e2e-llm-inference-service] conn = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Track whether `conn` needs to be released before [e2e-llm-inference-service] # returning/raising/recursing. Update this variable if necessary, and [e2e-llm-inference-service] # leave `release_conn` constant throughout the function. That way, if [e2e-llm-inference-service] # the function recurses, the original value of `release_conn` will be [e2e-llm-inference-service] # passed down into the recursive call, and its value will be respected. [e2e-llm-inference-service] # [e2e-llm-inference-service] # See issue #651 [1] for details. [e2e-llm-inference-service] # [e2e-llm-inference-service] # [1] [e2e-llm-inference-service] release_this_conn = release_conn [e2e-llm-inference-service] [e2e-llm-inference-service] http_tunnel_required = connection_requires_http_tunnel( [e2e-llm-inference-service] self.proxy, self.proxy_config, destination_scheme [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Merge the proxy headers. Only done when not using HTTP CONNECT. We [e2e-llm-inference-service] # have to copy the headers dict so we can safely change it without those [e2e-llm-inference-service] # changes being reflected in anyone else's copy. [e2e-llm-inference-service] if not http_tunnel_required: [e2e-llm-inference-service] headers = headers.copy() # type: ignore[attr-defined] [e2e-llm-inference-service] headers.update(self.proxy_headers) # type: ignore[union-attr] [e2e-llm-inference-service] [e2e-llm-inference-service] # Must keep the exception bound to a separate variable or else Python 3 [e2e-llm-inference-service] # complains about UnboundLocalError. [e2e-llm-inference-service] err = None [e2e-llm-inference-service] [e2e-llm-inference-service] # Keep track of whether we cleanly exited the except block. This [e2e-llm-inference-service] # ensures we do proper cleanup in finally. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] [e2e-llm-inference-service] # Rewind body position, if needed. Record current position [e2e-llm-inference-service] # for future rewinds in the event of a redirect/retry. [e2e-llm-inference-service] body_pos = set_file_position(body, body_pos) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Request a connection from the queue. [e2e-llm-inference-service] timeout_obj = self._get_timeout(timeout) [e2e-llm-inference-service] conn = self._get_conn(timeout=pool_timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] conn.timeout = timeout_obj.connect_timeout # type: ignore[assignment] [e2e-llm-inference-service] [e2e-llm-inference-service] # Is this a closed/new connection that requires CONNECT tunnelling? [e2e-llm-inference-service] if self.proxy is not None and http_tunnel_required and conn.is_closed: [e2e-llm-inference-service] try: [e2e-llm-inference-service] self._prepare_proxy(conn) [e2e-llm-inference-service] except (BaseSSLError, OSError, SocketTimeout) as e: [e2e-llm-inference-service] self._raise_timeout( [e2e-llm-inference-service] err=e, url=self.proxy.url, timeout_value=conn.timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] # If we're going to release the connection in ``finally:``, then [e2e-llm-inference-service] # the response doesn't need to know about the connection. Otherwise [e2e-llm-inference-service] # it will also try to release it and we'll have a double-release [e2e-llm-inference-service] # mess. [e2e-llm-inference-service] response_conn = conn if not release_conn else None [e2e-llm-inference-service] [e2e-llm-inference-service] # Make the request on the HTTPConnection object [e2e-llm-inference-service] response = self._make_request( [e2e-llm-inference-service] conn, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] timeout=timeout_obj, [e2e-llm-inference-service] body=body, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] retries=retries, [e2e-llm-inference-service] response_conn=response_conn, [e2e-llm-inference-service] preload_content=preload_content, [e2e-llm-inference-service] decode_content=decode_content, [e2e-llm-inference-service] **response_kw, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Everything went great! [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] [e2e-llm-inference-service] except EmptyPoolError: [e2e-llm-inference-service] # Didn't get a connection from the pool, no need to clean up [e2e-llm-inference-service] clean_exit = True [e2e-llm-inference-service] release_this_conn = False [e2e-llm-inference-service] raise [e2e-llm-inference-service] [e2e-llm-inference-service] except ( [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] ProtocolError, [e2e-llm-inference-service] BaseSSLError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] CertificateError, [e2e-llm-inference-service] ProxyError, [e2e-llm-inference-service] ) as e: [e2e-llm-inference-service] # Discard the connection for these exceptions. It will be [e2e-llm-inference-service] # replaced during the next _get_conn() call. [e2e-llm-inference-service] clean_exit = False [e2e-llm-inference-service] new_e: Exception = e [e2e-llm-inference-service] if isinstance(e, (BaseSSLError, CertificateError)): [e2e-llm-inference-service] new_e = SSLError(e) [e2e-llm-inference-service] if isinstance( [e2e-llm-inference-service] new_e, [e2e-llm-inference-service] ( [e2e-llm-inference-service] OSError, [e2e-llm-inference-service] NewConnectionError, [e2e-llm-inference-service] TimeoutError, [e2e-llm-inference-service] SSLError, [e2e-llm-inference-service] HTTPException, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ) and (conn and conn.proxy and not conn.has_connected_to_proxy): [e2e-llm-inference-service] new_e = _wrap_proxy_error(new_e, conn.proxy.scheme) [e2e-llm-inference-service] elif isinstance(new_e, (OSError, HTTPException)): [e2e-llm-inference-service] new_e = ProtocolError("Connection aborted.", new_e) [e2e-llm-inference-service] [e2e-llm-inference-service] > retries = retries.increment( [e2e-llm-inference-service] method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/connectionpool.py:841: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = Retry(total=0, connect=None, read=None, redirect=None, status=None) [e2e-llm-inference-service] method = 'POST' [e2e-llm-inference-service] url = '/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions' [e2e-llm-inference-service] response = None [e2e-llm-inference-service] error = ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)") [e2e-llm-inference-service] _pool = [e2e-llm-inference-service] _stacktrace = [e2e-llm-inference-service] [e2e-llm-inference-service] def increment( [e2e-llm-inference-service] self, [e2e-llm-inference-service] method: str | None = None, [e2e-llm-inference-service] url: str | None = None, [e2e-llm-inference-service] response: BaseHTTPResponse | None = None, [e2e-llm-inference-service] error: Exception | None = None, [e2e-llm-inference-service] _pool: ConnectionPool | None = None, [e2e-llm-inference-service] _stacktrace: TracebackType | None = None, [e2e-llm-inference-service] ) -> Self: [e2e-llm-inference-service] """Return a new Retry object with incremented retry counters. [e2e-llm-inference-service] [e2e-llm-inference-service] :param response: A response object, or None, if the server did not [e2e-llm-inference-service] return a response. [e2e-llm-inference-service] :type response: :class:`~urllib3.response.BaseHTTPResponse` [e2e-llm-inference-service] :param Exception error: An error encountered during the request, or [e2e-llm-inference-service] None if the response was received successfully. [e2e-llm-inference-service] [e2e-llm-inference-service] :return: A new ``Retry`` object. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if self.total is False and error: [e2e-llm-inference-service] # Disabled, indicate to re-raise the error. [e2e-llm-inference-service] raise reraise(type(error), error, _stacktrace) [e2e-llm-inference-service] [e2e-llm-inference-service] total = self.total [e2e-llm-inference-service] if total is not None: [e2e-llm-inference-service] total -= 1 [e2e-llm-inference-service] [e2e-llm-inference-service] connect = self.connect [e2e-llm-inference-service] read = self.read [e2e-llm-inference-service] redirect = self.redirect [e2e-llm-inference-service] status_count = self.status [e2e-llm-inference-service] other = self.other [e2e-llm-inference-service] cause = "unknown" [e2e-llm-inference-service] status = None [e2e-llm-inference-service] redirect_location = None [e2e-llm-inference-service] [e2e-llm-inference-service] if error and self._is_connection_error(error): [e2e-llm-inference-service] # Connect retry? [e2e-llm-inference-service] if connect is False: [e2e-llm-inference-service] raise reraise(type(error), error, _stacktrace) [e2e-llm-inference-service] elif connect is not None: [e2e-llm-inference-service] connect -= 1 [e2e-llm-inference-service] [e2e-llm-inference-service] elif error and self._is_read_error(error): [e2e-llm-inference-service] # Read retry? [e2e-llm-inference-service] if read is False or method is None or not self._is_method_retryable(method): [e2e-llm-inference-service] raise reraise(type(error), error, _stacktrace) [e2e-llm-inference-service] elif read is not None: [e2e-llm-inference-service] read -= 1 [e2e-llm-inference-service] [e2e-llm-inference-service] elif error: [e2e-llm-inference-service] # Other retry? [e2e-llm-inference-service] if other is not None: [e2e-llm-inference-service] other -= 1 [e2e-llm-inference-service] [e2e-llm-inference-service] elif response and response.get_redirect_location(): [e2e-llm-inference-service] # Redirect retry? [e2e-llm-inference-service] if redirect is not None: [e2e-llm-inference-service] redirect -= 1 [e2e-llm-inference-service] cause = "too many redirects" [e2e-llm-inference-service] response_redirect_location = response.get_redirect_location() [e2e-llm-inference-service] if response_redirect_location: [e2e-llm-inference-service] redirect_location = response_redirect_location [e2e-llm-inference-service] status = response.status [e2e-llm-inference-service] [e2e-llm-inference-service] else: [e2e-llm-inference-service] # Incrementing because of a server error like a 500 in [e2e-llm-inference-service] # status_forcelist and the given method is in the allowed_methods [e2e-llm-inference-service] cause = ResponseError.GENERIC_ERROR [e2e-llm-inference-service] if response and response.status: [e2e-llm-inference-service] if status_count is not None: [e2e-llm-inference-service] status_count -= 1 [e2e-llm-inference-service] cause = ResponseError.SPECIFIC_ERROR.format(status_code=response.status) [e2e-llm-inference-service] status = response.status [e2e-llm-inference-service] [e2e-llm-inference-service] history = self.history + ( [e2e-llm-inference-service] RequestHistory(method, url, error, status, redirect_location), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] new_retry = self.new( [e2e-llm-inference-service] total=total, [e2e-llm-inference-service] connect=connect, [e2e-llm-inference-service] read=read, [e2e-llm-inference-service] redirect=redirect, [e2e-llm-inference-service] status=status_count, [e2e-llm-inference-service] other=other, [e2e-llm-inference-service] history=history, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if new_retry.is_exhausted(): [e2e-llm-inference-service] reason = error or ResponseError(cause) [e2e-llm-inference-service] > raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] [e2e-llm-inference-service] E urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions (Caused by ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/urllib3/util/retry.py:519: MaxRetryError [e2e-llm-inference-service] [e2e-llm-inference-service] During handling of the above exception, another exception occurred: [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = LoRATestCase(base_refs=['router-no-scheduler', 'workload-single-cpu', 'model-fb-opt-125m-with-multiple-lora'], prompt=..._name='lora-multiple-adapters-test', endpoint='/v1/completions', max_tokens=100, wait_timeout=900, response_timeout=60) [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] LoRATestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is Kubernetes?", [e2e-llm-inference-service] expected_adapter_names=["lora-adapter-1"], [e2e-llm-inference-service] service_name="lora-single-adapter-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.llminferenceservice, [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] id="single-lora-adapter-hf", [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] LoRATestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-multiple-lora", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Explain machine learning in simple terms.", [e2e-llm-inference-service] expected_adapter_names=["lora-adapter-1", "lora-adapter-2"], [e2e-llm-inference-service] service_name="lora-multiple-adapters-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.llminferenceservice, [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] id="multiple-lora-adapters", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_with_lora_adapters(test_case: LoRATestCase): [e2e-llm-inference-service] """Test LLMInferenceService with LoRA adapters.""" [e2e-llm-inference-service] > run_lora_test(test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py:248: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = LoRATestCase(base_refs=['router-no-scheduler', 'workload-single-cpu', 'model-fb-opt-125m-with-multiple-lora'], prompt=..._name='lora-multiple-adapters-test', endpoint='/v1/completions', max_tokens=100, wait_timeout=900, response_timeout=60) [e2e-llm-inference-service] [e2e-llm-inference-service] def run_lora_test(test_case: LoRATestCase): [e2e-llm-inference-service] """Execute a LoRA adapter test case.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] test_id = generate_test_id(test_case) [e2e-llm-inference-service] service_name = test_case.service_name or f"lora-test-{test_id}" [e2e-llm-inference-service] model_name = _get_model_name_from_configs(test_case.base_refs) [e2e-llm-inference-service] created_configs = [] [e2e-llm-inference-service] llm_service = None [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] # Create unique LLMInferenceServiceConfig resources for each base ref [e2e-llm-inference-service] unique_base_refs = [] [e2e-llm-inference-service] for base_ref in test_case.base_refs: [e2e-llm-inference-service] if base_ref not in LLMINFERENCESERVICE_CONFIGS: [e2e-llm-inference-service] raise ValueError(f"Unknown base reference: {base_ref}") [e2e-llm-inference-service] [e2e-llm-inference-service] # Generate unique config name to avoid conflicts in parallel test runs [e2e-llm-inference-service] unique_config_name = generate_k8s_safe_suffix(base_ref, [service_name]) [e2e-llm-inference-service] unique_base_refs.append(unique_config_name) [e2e-llm-inference-service] [e2e-llm-inference-service] config = LLMINFERENCESERVICE_CONFIGS[base_ref] [e2e-llm-inference-service] config_body = { [e2e-llm-inference-service] "apiVersion": "serving.kserve.io/v1alpha1", [e2e-llm-inference-service] "kind": "LLMInferenceServiceConfig", [e2e-llm-inference-service] "metadata": { [e2e-llm-inference-service] "name": unique_config_name, [e2e-llm-inference-service] "namespace": KSERVE_TEST_NAMESPACE, [e2e-llm-inference-service] }, [e2e-llm-inference-service] "spec": config, [e2e-llm-inference-service] } [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info("Creating LLMInferenceServiceConfig: %s", unique_config_name) [e2e-llm-inference-service] _create_or_update_llmisvc_config( [e2e-llm-inference-service] kserve_client, config_body, KSERVE_TEST_NAMESPACE [e2e-llm-inference-service] ) [e2e-llm-inference-service] created_configs.append(unique_config_name) [e2e-llm-inference-service] [e2e-llm-inference-service] # Create the service with unique base refs [e2e-llm-inference-service] llm_service = build_llm_service_from_refs(service_name, unique_base_refs) [e2e-llm-inference-service] if not llm_service.metadata.annotations: [e2e-llm-inference-service] llm_service.metadata.annotations = {} [e2e-llm-inference-service] llm_service.metadata.annotations["security.opendatahub.io/enable-auth"] = ( [e2e-llm-inference-service] "false" [e2e-llm-inference-service] ) [e2e-llm-inference-service] logger.info("Creating LLMInferenceService: %s", service_name) [e2e-llm-inference-service] create_llmisvc(kserve_client, llm_service) [e2e-llm-inference-service] [e2e-llm-inference-service] # Wait for service to be ready [e2e-llm-inference-service] logger.info("Waiting for service %s to be ready...", service_name) [e2e-llm-inference-service] wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, llm_service, timeout_seconds=test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Get inference URL [e2e-llm-inference-service] base_url = get_llm_service_url(kserve_client, llm_service) [e2e-llm-inference-service] inference_url = base_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] # Test base model inference [e2e-llm-inference-service] logger.info("Testing base model inference...") [e2e-llm-inference-service] base_payload = { [e2e-llm-inference-service] "model": model_name, [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] [e2e-llm-inference-service] > base_response = post_with_retry( [e2e-llm-inference-service] inference_url, [e2e-llm-inference-service] json_data=base_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py:150: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] url = 'http://a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions' [e2e-llm-inference-service] [e2e-llm-inference-service] def post_with_retry( [e2e-llm-inference-service] url: str, [e2e-llm-inference-service] *, [e2e-llm-inference-service] headers: Dict = None, [e2e-llm-inference-service] json_data: Union[Dict, List] = None, [e2e-llm-inference-service] data: Union[str, bytes] = None, [e2e-llm-inference-service] stream: bool = False, [e2e-llm-inference-service] timeout: float = None, [e2e-llm-inference-service] total_retries: int = DEFAULT_RETRY_TOTAL, [e2e-llm-inference-service] backoff_factor: float = DEFAULT_RETRY_BACKOFF_FACTOR, [e2e-llm-inference-service] retry_status_codes=DEFAULT_RETRY_STATUS_CODES, [e2e-llm-inference-service] ) -> requests.Response: [e2e-llm-inference-service] """ [e2e-llm-inference-service] Send POST request with retries for transient HTTP and network failures. [e2e-llm-inference-service] """ [e2e-llm-inference-service] if json_data is not None and data is not None: [e2e-llm-inference-service] raise ValueError("Only one of json_data or data can be provided.") [e2e-llm-inference-service] [e2e-llm-inference-service] with _retry_session( [e2e-llm-inference-service] ["POST"], total_retries, backoff_factor, retry_status_codes [e2e-llm-inference-service] ) as session: [e2e-llm-inference-service] > return session.post( [e2e-llm-inference-service] url, [e2e-llm-inference-service] json=json_data, [e2e-llm-inference-service] data=data, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] stream=stream, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] common/http_retry.py:70: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] url = 'http://a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions' [e2e-llm-inference-service] data = None [e2e-llm-inference-service] json = {'max_tokens': 100, 'model': 'facebook/opt-125m', 'prompt': 'Explain machine learning in simple terms.'} [e2e-llm-inference-service] kwargs = {'headers': None, 'stream': False, 'timeout': 60} [e2e-llm-inference-service] [e2e-llm-inference-service] def post(self, url, data=None, json=None, **kwargs): [e2e-llm-inference-service] r"""Sends a POST request. Returns :class:`Response` object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param url: URL for the new :class:`Request` object. [e2e-llm-inference-service] :param data: (optional) Dictionary, list of tuples, bytes, or file-like [e2e-llm-inference-service] object to send in the body of the :class:`Request`. [e2e-llm-inference-service] :param json: (optional) json to send in the body of the :class:`Request`. [e2e-llm-inference-service] :param \*\*kwargs: Optional arguments that ``request`` takes. [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] > return self.request("POST", url, data=data, json=json, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/sessions.py:637: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = , method = 'POST' [e2e-llm-inference-service] url = 'http://a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions' [e2e-llm-inference-service] params = None, data = None, headers = None, cookies = None, files = None [e2e-llm-inference-service] auth = None, timeout = 60, allow_redirects = True, proxies = {}, hooks = None [e2e-llm-inference-service] stream = False, verify = None, cert = None [e2e-llm-inference-service] json = {'max_tokens': 100, 'model': 'facebook/opt-125m', 'prompt': 'Explain machine learning in simple terms.'} [e2e-llm-inference-service] [e2e-llm-inference-service] def request( [e2e-llm-inference-service] self, [e2e-llm-inference-service] method, [e2e-llm-inference-service] url, [e2e-llm-inference-service] params=None, [e2e-llm-inference-service] data=None, [e2e-llm-inference-service] headers=None, [e2e-llm-inference-service] cookies=None, [e2e-llm-inference-service] files=None, [e2e-llm-inference-service] auth=None, [e2e-llm-inference-service] timeout=None, [e2e-llm-inference-service] allow_redirects=True, [e2e-llm-inference-service] proxies=None, [e2e-llm-inference-service] hooks=None, [e2e-llm-inference-service] stream=None, [e2e-llm-inference-service] verify=None, [e2e-llm-inference-service] cert=None, [e2e-llm-inference-service] json=None, [e2e-llm-inference-service] ): [e2e-llm-inference-service] """Constructs a :class:`Request `, prepares it and sends it. [e2e-llm-inference-service] Returns :class:`Response ` object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param method: method for the new :class:`Request` object. [e2e-llm-inference-service] :param url: URL for the new :class:`Request` object. [e2e-llm-inference-service] :param params: (optional) Dictionary or bytes to be sent in the query [e2e-llm-inference-service] string for the :class:`Request`. [e2e-llm-inference-service] :param data: (optional) Dictionary, list of tuples, bytes, or file-like [e2e-llm-inference-service] object to send in the body of the :class:`Request`. [e2e-llm-inference-service] :param json: (optional) json to send in the body of the [e2e-llm-inference-service] :class:`Request`. [e2e-llm-inference-service] :param headers: (optional) Dictionary of HTTP Headers to send with the [e2e-llm-inference-service] :class:`Request`. [e2e-llm-inference-service] :param cookies: (optional) Dict or CookieJar object to send with the [e2e-llm-inference-service] :class:`Request`. [e2e-llm-inference-service] :param files: (optional) Dictionary of ``'filename': file-like-objects`` [e2e-llm-inference-service] for multipart encoding upload. [e2e-llm-inference-service] :param auth: (optional) Auth tuple or callable to enable [e2e-llm-inference-service] Basic/Digest/Custom HTTP Auth. [e2e-llm-inference-service] :param timeout: (optional) How long to wait for the server to send [e2e-llm-inference-service] data before giving up, as a float, or a :ref:`(connect timeout, [e2e-llm-inference-service] read timeout) ` tuple. [e2e-llm-inference-service] :type timeout: float or tuple [e2e-llm-inference-service] :param allow_redirects: (optional) Set to True by default. [e2e-llm-inference-service] :type allow_redirects: bool [e2e-llm-inference-service] :param proxies: (optional) Dictionary mapping protocol or protocol and [e2e-llm-inference-service] hostname to the URL of the proxy. [e2e-llm-inference-service] :param hooks: (optional) Dictionary mapping hook name to one event or [e2e-llm-inference-service] list of events, event must be callable. [e2e-llm-inference-service] :param stream: (optional) whether to immediately download the response [e2e-llm-inference-service] content. Defaults to ``False``. [e2e-llm-inference-service] :param verify: (optional) Either a boolean, in which case it controls whether we verify [e2e-llm-inference-service] the server's TLS certificate, or a string, in which case it must be a path [e2e-llm-inference-service] to a CA bundle to use. Defaults to ``True``. When set to [e2e-llm-inference-service] ``False``, requests will accept any TLS certificate presented by [e2e-llm-inference-service] the server, and will ignore hostname mismatches and/or expired [e2e-llm-inference-service] certificates, which will make your application vulnerable to [e2e-llm-inference-service] man-in-the-middle (MitM) attacks. Setting verify to ``False`` [e2e-llm-inference-service] may be useful during local development or testing. [e2e-llm-inference-service] :param cert: (optional) if String, path to ssl client cert file (.pem). [e2e-llm-inference-service] If Tuple, ('cert', 'key') pair. [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] # Create the Request. [e2e-llm-inference-service] req = Request( [e2e-llm-inference-service] method=method.upper(), [e2e-llm-inference-service] url=url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] files=files, [e2e-llm-inference-service] data=data or {}, [e2e-llm-inference-service] json=json, [e2e-llm-inference-service] params=params or {}, [e2e-llm-inference-service] auth=auth, [e2e-llm-inference-service] cookies=cookies, [e2e-llm-inference-service] hooks=hooks, [e2e-llm-inference-service] ) [e2e-llm-inference-service] prep = self.prepare_request(req) [e2e-llm-inference-service] [e2e-llm-inference-service] proxies = proxies or {} [e2e-llm-inference-service] [e2e-llm-inference-service] settings = self.merge_environment_settings( [e2e-llm-inference-service] prep.url, proxies, stream, verify, cert [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] # Send the request. [e2e-llm-inference-service] send_kwargs = { [e2e-llm-inference-service] "timeout": timeout, [e2e-llm-inference-service] "allow_redirects": allow_redirects, [e2e-llm-inference-service] } [e2e-llm-inference-service] send_kwargs.update(settings) [e2e-llm-inference-service] > resp = self.send(prep, **send_kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/sessions.py:589: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] request = [e2e-llm-inference-service] kwargs = {'cert': None, 'proxies': OrderedDict(), 'stream': False, 'timeout': 60, ...} [e2e-llm-inference-service] allow_redirects = True, stream = False, hooks = {'response': []} [e2e-llm-inference-service] adapter = [e2e-llm-inference-service] start = 1783373808.9848943 [e2e-llm-inference-service] [e2e-llm-inference-service] def send(self, request, **kwargs): [e2e-llm-inference-service] """Send a given PreparedRequest. [e2e-llm-inference-service] [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] # Set defaults that the hooks can utilize to ensure they always have [e2e-llm-inference-service] # the correct parameters to reproduce the previous request. [e2e-llm-inference-service] kwargs.setdefault("stream", self.stream) [e2e-llm-inference-service] kwargs.setdefault("verify", self.verify) [e2e-llm-inference-service] kwargs.setdefault("cert", self.cert) [e2e-llm-inference-service] if "proxies" not in kwargs: [e2e-llm-inference-service] kwargs["proxies"] = resolve_proxies(request, self.proxies, self.trust_env) [e2e-llm-inference-service] [e2e-llm-inference-service] # It's possible that users might accidentally send a Request object. [e2e-llm-inference-service] # Guard against that specific failure case. [e2e-llm-inference-service] if isinstance(request, Request): [e2e-llm-inference-service] raise ValueError("You can only send PreparedRequests.") [e2e-llm-inference-service] [e2e-llm-inference-service] # Set up variables needed for resolve_redirects and dispatching of hooks [e2e-llm-inference-service] allow_redirects = kwargs.pop("allow_redirects", True) [e2e-llm-inference-service] stream = kwargs.get("stream") [e2e-llm-inference-service] hooks = request.hooks [e2e-llm-inference-service] [e2e-llm-inference-service] # Get the appropriate adapter to use [e2e-llm-inference-service] adapter = self.get_adapter(url=request.url) [e2e-llm-inference-service] [e2e-llm-inference-service] # Start time (approximately) of the request [e2e-llm-inference-service] start = preferred_clock() [e2e-llm-inference-service] [e2e-llm-inference-service] # Send the request [e2e-llm-inference-service] > r = adapter.send(request, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/sessions.py:703: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] self = [e2e-llm-inference-service] request = , stream = False [e2e-llm-inference-service] timeout = Timeout(connect=60, read=60, total=None), verify = '/tmp/ca.crt' [e2e-llm-inference-service] cert = None, proxies = OrderedDict() [e2e-llm-inference-service] [e2e-llm-inference-service] def send( [e2e-llm-inference-service] self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None [e2e-llm-inference-service] ): [e2e-llm-inference-service] """Sends PreparedRequest object. Returns Response object. [e2e-llm-inference-service] [e2e-llm-inference-service] :param request: The :class:`PreparedRequest ` being sent. [e2e-llm-inference-service] :param stream: (optional) Whether to stream the request content. [e2e-llm-inference-service] :param timeout: (optional) How long to wait for the server to send [e2e-llm-inference-service] data before giving up, as a float, or a :ref:`(connect timeout, [e2e-llm-inference-service] read timeout) ` tuple. [e2e-llm-inference-service] :type timeout: float or tuple or urllib3 Timeout object [e2e-llm-inference-service] :param verify: (optional) Either a boolean, in which case it controls whether [e2e-llm-inference-service] we verify the server's TLS certificate, or a string, in which case it [e2e-llm-inference-service] must be a path to a CA bundle to use [e2e-llm-inference-service] :param cert: (optional) Any user-provided SSL certificate to be trusted. [e2e-llm-inference-service] :param proxies: (optional) The proxies dictionary to apply to the request. [e2e-llm-inference-service] :rtype: requests.Response [e2e-llm-inference-service] """ [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] conn = self.get_connection_with_tls_context( [e2e-llm-inference-service] request, verify, proxies=proxies, cert=cert [e2e-llm-inference-service] ) [e2e-llm-inference-service] except LocationValueError as e: [e2e-llm-inference-service] raise InvalidURL(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] self.cert_verify(conn, request.url, verify, cert) [e2e-llm-inference-service] url = self.request_url(request, proxies) [e2e-llm-inference-service] self.add_headers( [e2e-llm-inference-service] request, [e2e-llm-inference-service] stream=stream, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] verify=verify, [e2e-llm-inference-service] cert=cert, [e2e-llm-inference-service] proxies=proxies, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] chunked = not (request.body is None or "Content-Length" in request.headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(timeout, tuple): [e2e-llm-inference-service] try: [e2e-llm-inference-service] connect, read = timeout [e2e-llm-inference-service] timeout = TimeoutSauce(connect=connect, read=read) [e2e-llm-inference-service] except ValueError: [e2e-llm-inference-service] raise ValueError( [e2e-llm-inference-service] f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, " [e2e-llm-inference-service] f"or a single float to set both timeouts to the same value." [e2e-llm-inference-service] ) [e2e-llm-inference-service] elif isinstance(timeout, TimeoutSauce): [e2e-llm-inference-service] pass [e2e-llm-inference-service] else: [e2e-llm-inference-service] timeout = TimeoutSauce(connect=timeout, read=timeout) [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] resp = conn.urlopen( [e2e-llm-inference-service] method=request.method, [e2e-llm-inference-service] url=url, [e2e-llm-inference-service] body=request.body, [e2e-llm-inference-service] headers=request.headers, [e2e-llm-inference-service] redirect=False, [e2e-llm-inference-service] assert_same_host=False, [e2e-llm-inference-service] preload_content=False, [e2e-llm-inference-service] decode_content=False, [e2e-llm-inference-service] retries=self.max_retries, [e2e-llm-inference-service] timeout=timeout, [e2e-llm-inference-service] chunked=chunked, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] except (ProtocolError, OSError) as err: [e2e-llm-inference-service] raise ConnectionError(err, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] except MaxRetryError as e: [e2e-llm-inference-service] if isinstance(e.reason, ConnectTimeoutError): [e2e-llm-inference-service] # TODO: Remove this in 3.0.0: see #2811 [e2e-llm-inference-service] if not isinstance(e.reason, NewConnectionError): [e2e-llm-inference-service] raise ConnectTimeout(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(e.reason, ResponseError): [e2e-llm-inference-service] raise RetryError(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(e.reason, _ProxyError): [e2e-llm-inference-service] raise ProxyError(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] if isinstance(e.reason, _SSLError): [e2e-llm-inference-service] # This branch is for urllib3 v1.22 and later. [e2e-llm-inference-service] raise SSLError(e, request=request) [e2e-llm-inference-service] [e2e-llm-inference-service] > raise ConnectionError(e, request=request) [e2e-llm-inference-service] E requests.exceptions.ConnectionError: HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions (Caused by ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")) [e2e-llm-inference-service] [e2e-llm-inference-service] ../../python/kserve/.venv/lib64/python3.11/site-packages/requests/adapters.py:700: ConnectionError [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_with_lora_adapters] [2026-07-06T21:35:08.511657] start - args=(), kwargs={'test_case': LoRATestCase(base_refs=['router-no-scheduler', 'workload-single-cpu', 'model-fb-opt-125m-with-multiple-lora'], prompt='Explain machine learning in simple terms.', expected_adapter_names=['lora-adapter-1', 'lora-adapter-2'], service_name='lora-multiple-adapters-test', endpoint='/v1/completions', max_tokens=100, wait_timeout=900, response_timeout=60)} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_lora_adapters:test_llm_lora_adapters.py:116 Creating LLMInferenceServiceConfig: router-no-scheduler-lora-multip-343dff1f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig router-no-scheduler-lora-multip-343dff1f in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig router-no-scheduler-lora-multip-343dff1f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig router-no-scheduler-lora-multip-343dff1f [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_lora_adapters:test_llm_lora_adapters.py:116 Creating LLMInferenceServiceConfig: workload-single-cpu-lora-multip-b14d2ed8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig workload-single-cpu-lora-multip-b14d2ed8 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig workload-single-cpu-lora-multip-b14d2ed8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig workload-single-cpu-lora-multip-b14d2ed8 [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_lora_adapters:test_llm_lora_adapters.py:116 Creating LLMInferenceServiceConfig: model-fb-opt-125m-with-multiple-7cbd2065 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig model-fb-opt-125m-with-multiple-7cbd2065 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig model-fb-opt-125m-with-multiple-7cbd2065 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig model-fb-opt-125m-with-multiple-7cbd2065 [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_lora_adapters:test_llm_lora_adapters.py:129 Creating LLMInferenceService: lora-multiple-adapters-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-06T21:35:08.618443] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'lora-multiple-adapters-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-no-scheduler-lora-multip-343dff1f'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-lora-multip-b14d2ed8'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-multiple-7cbd2065'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-06T21:35:08.674672] end - ✅ in 0.056s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_lora_adapters:test_llm_lora_adapters.py:133 Waiting for service lora-multiple-adapters-test to be ready... [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-06T21:35:08.674950] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'lora-multiple-adapters-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-no-scheduler-lora-multip-343dff1f'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-lora-multip-b14d2ed8'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-multiple-7cbd2065'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={'timeout_seconds': 900} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-06T21:35:18Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T21:35:18Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:35:18Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T21:35:18Z', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T21:35:18Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T21:35:18Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-06T21:35:18Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T21:35:25Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:35:18Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T21:35:25Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T21:35:18Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T21:35:25Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_ready] [2026-07-06T21:36:48.973086] end - ✅ in 100.298s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-06T21:36:48.973239] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'lora-multiple-adapters-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-no-scheduler-lora-multip-343dff1f'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-lora-multip-b14d2ed8'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-multiple-7cbd2065'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-06T21:36:48.983656] end - ✅ in 0.010s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_lora_adapters:test_llm_lora_adapters.py:143 Testing base model inference... [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")': /kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")': /kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")': /kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")': /kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")': /kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")': /kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")': /kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")': /kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_lora_adapters:test_llm_lora_adapters.py:181 Cleaning up service lora-multiple-adapters-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-06T21:51:53.607292] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'lora-multiple-adapters-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-no-scheduler-lora-multip-343dff1f'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-lora-multip-b14d2ed8'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-multiple-7cbd2065'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: 1 pod(s) for lora-multiple-adapters-test still terminating [e2e-llm-inference-service] assert not [{'api_version': None,\n 'kind': None,\n 'metadata': {'annotations': {'k8s.ovn.org/pod-networks': '{"default":{"ip_addresses":["10.134.0.76/23"],"mac_address":"0a:58:0a:86:00:4c","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.76/23","gateway_ip":"10.134.0.1","role":"primary"}}',\n 'k8s.v1.cni.cncf.io/network-status': '[{\n'\n ' '\n '"name": '\n '"ovn-kubernetes",\n'\n ' '\n '"interface": '\n '"eth0",\n'\n ' "ips": '\n '[\n'\n ' '\n '"10.134.0.76"\n'\n ' ],\n'\n ' "mac": '\n '"0a:58:0a:86:00:4c",\n'\n ' '\n '"default": '\n 'true,\n'\n ' "dns": '\n '{}\n'\n '}]',\n 'openshift.io/scc': 'restricted-v2',\n 'seccomp.security.alpha.kubernetes.io/pod': 'runtime/default',\n 'security.openshift.io/validated-scc-subject-type': 'user'},\n 'creation_timestamp': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'deletion_grace_period_seconds': None,\n 'deletion_timestamp': None,\n 'finalizers': None,\n 'generate_name': 'lora-multiple-adapters-test-kserve-77485bff95-',\n 'generation': 1,\n 'labels': {'app.kubernetes.io/component': 'llminferenceservice-workload',\n 'app.kubernetes.io/name': 'lora-multiple-adapters-test',\n 'app.kubernetes.io/part-of': 'llminferenceservice',\n 'kserve.io/component': 'workload',\n 'llm-d.ai/role': 'both',\n 'pod-template-hash': '77485bff95'},\n 'managed_fields': [{'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.ovn.org/pod-networks': {}}}},\n 'manager': 'ip-10-0-131-57',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:generateName': {},\n 'f:labels': {'.': {},\n 'f:app.kubernetes.io/component': {},\n [e2e-llm-inference-service] 'f:app.kubernetes.io/name': {},\n 'f:app.kubernetes.io/part-of': {},\n 'f:kserve.io/component': {},\n 'f:llm-d.ai/role': {},\n 'f:pod-template-hash': {}},\n 'f:ownerReferences': {'.': {},\n 'k:{"uid":"75c381c4-1742-4fb9-b05f-92ea6db35c9c"}': {}}},\n 'f:spec': {'f:containers': {'k:{"name":"main"}': {'.': {},\n 'f:args': {},\n 'f:command': {},\n 'f:env': {'.': {},\n 'k:{"name":"HF_HUB_CACHE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HOME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"TORCHINDUCTOR_CACHE_DIR"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"USER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_CPU_KVCACHE_SPACE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k: [e2e-llm-inference-service] {"name":"VLLM_LOGGING_LEVEL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:lifecycle': {'.': {},\n 'f:preStop': {'.': {},\n 'f:exec': {'.': {},\n 'f:command': {}}}},\n 'f:livenessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:name': {},\n 'f:ports': {'.': {},\n 'k:{"containerPort":8000,"protocol":"TCP"}': {'.': {},\n 'f:containerPort': {},\n 'f:protocol': {}}},\n 'f:readinessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n [e2e-llm-inference-service] 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:securityContext': {'.': {},\n 'f:allowPrivilegeEscalation': {},\n 'f:capabilities': {'.': {},\n 'f:drop': {}},\n 'f:readOnlyRootFilesystem': {},\n 'f:runAsNonRoot': {},\n 'f:seccompProfile': {'.': {},\n 'f:type': {}}},\n 'f:startupProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:terminationMessagePath': {},\n [e2e-llm-inference-service] 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/dev/shm"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/home"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/models"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/tmp"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/var/run/kserve/tls"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}}}}},\n 'f:dnsPolicy': {},\n 'f:enableServiceLinks': {},\n 'f:initContainers': {'.': {},\n 'k:{"name":"storage-initializer"}': {'.': {},\n 'f:args': {},\n 'f:env': {'.' [e2e-llm-inference-service] : {},\n 'k:{"name":"AWS_ACCESS_KEY_ID"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"AWS_CA_BUNDLE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_ENDPOINT_URL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_SECRET_ACCESS_KEY"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}': {'.': {},\n 'f:name': {},\n [e2e-llm-inference-service] 'f:value': {}},\n 'k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_HIGH_PERFORMANCE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_ENDPOINT"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_USE_HTTPS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_VERIFY_SSL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:name': {},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n [e2e-llm-inference-service] 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:terminationMessagePath': {},\n 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/etc/ssl/custom-certs"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}}}}},\n 'f:restartPolicy': {},\n 'f:schedulerName': {},\n 'f:securityContext': {},\n 'f:terminationGracePeriodSeconds': {},\n 'f:volumes': {'.': {},\n 'k:{"name":"cabundle-cert"}': {'.': {},\n 'f:configMap': {'.': {},\n 'f:defaultMode': {},\n 'f:name': {}},\n 'f:name': {}},\n 'k:{"name":"dshm"}': {'.': {},\n 'f:emptyDir': {'.': {},\n 'f:medium': {},\n 'f:sizeLimit': {}},\n [e2e-llm-inference-service] 'f:name': {}},\n 'k:{"name":"home"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"kserve-provision-location"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"model-cache"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"tls-certs"}': {'.': {},\n 'f:name': {},\n 'f:secret': {'.': {},\n 'f:defaultMode': {},\n 'f:secretName': {}}},\n 'k:{"name":"tmp-dir"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}}}}},\n 'manager': 'kube-controller-manager',\n 'operation': 'Update',\n 'subresource': None,\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.v1.cni.cncf.io/network-status': {}}}},\n 'manager': 'multus-daemon',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:status': {'f:conditions': {'k:{"type":"ContainersReady"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"Initialized"}': {'.': {},\n 'f:lastProbeTime': {},\ [e2e-llm-inference-service] n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodReadyToStartContainers"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodScheduled"}': {'f:observedGeneration': {}},\n 'k:{"type":"Ready"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}}},\n 'f:containerStatuses': {},\n 'f:hostIP': {},\n 'f:hostIPs': {},\n 'f:initContainerStatuses': {},\n 'f:observedGeneration': {},\n 'f:phase': {},\n 'f:podIP': {},\n 'f:podIPs': {'.': {},\n 'k:{"ip":"10.134.0.76"}': {'.': {},\n 'f:ip': {}}},\n 'f:startTime': {}}},\n 'manager': 'kubelet',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal())}],\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95-2tx5t',\n 'namespace': 'kserve-ci-e2e-test',\n 'owner_references': [{'api_version': 'apps/v1',\n 'block_owner_deletion': True,\n 'controller': True,\n 'kind': 'ReplicaSet',\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95',\n 'uid': '75c381c4-1742-4fb9-b05f-92ea6db35c9c'}],\n 'resource_version': '64798',\ [e2e-llm-inference-service] n 'self_link': None,\n 'uid': 'd4948f86-f0d1-4911-908f-7cb88db94468'},\n 'spec': {'active_deadline_seconds': None,\n 'affinity': None,\n 'automount_service_account_token': None,\n 'containers': [{'args': ['--enable-lora',\n '--max-lora-rank=64',\n '--max-loras=2',\n '--max-cpu-loras=4',\n '--lora-modules',\n '\'{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\''],\n 'command': ['/bin/bash',\n '-c',\n 'if [ -f /etc/profile.d/ibm-aiu-setup.sh '\n ']; then\n'\n ' source '\n '/etc/profile.d/ibm-aiu-setup.sh\n'\n 'fi\n'\n '\n'\n 'if [ "$KSERVE_INFER_ROCE" = "true" ]; '\n 'then\n'\n ' echo "Trying to infer RoCE configs '\n '... "\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gids/* '\n '2>/dev/null\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gid_attrs/types/* '\n '2>/dev/null\n'\n '\n'\n ' cat /proc/driver/nvidia/params\n'\n '\n'\n ' '\n 'KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE '\n 'v2"}\n'\n '\n'\n ' echo "[Infer RoCE] Discovering active '\n 'HCAs ..."\n'\n ' active_hcas=()\n'\n ' # Loop through all mlx5 devices found '\n 'in sysfs\n'\n ' for hca_dir in '\n '/sys/class/infiniband/mlx5_*; do\n'\n " # Ensure it's a directory before "\n 'proceeding\n'\n ' if [ -d "$hca_dir" ]; then\n'\n ' hca_name=$(basename '\n '"$hca_dir")\n'\n ' '\n 'port_state_file="$hca_dir/ports/1/state" '\n '# Assume port 1\n'\n ' '\n 'type_file="$hca_dir/ports/1/gid_attrs/types/*"\n'\n '\n'\n ' echo "[Infer RoCE] Check if '\n 'the port state file ${port_state_file} '\n 'exists and contains \'ACTIVE\'"\n'\n ' if [ -f "$port_state_file" ] '\n '&& grep -q "ACTIVE" "$port_state_file" '\n [e2e-llm-inference-service] '&& grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '${type_file} 2>/dev/null; then\n'\n ' echo "[Infer RoCE] Found '\n 'active HCA: $hca_name"\n'\n ' '\n 'active_hcas+=("$hca_name")\n'\n ' else\n'\n ' echo "[Infer RoCE] '\n 'Skipping inactive or down HCA: '\n '$hca_name"\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Check if we found any active HCAs\n'\n ' if [ ${#active_hcas[@]} -gt 0 ]; '\n 'then\n'\n ' # Join the array elements with a '\n 'comma\n'\n ' hca_port_pairs=()\n'\n ' for hca in "${active_hcas[@]}"; '\n 'do\n'\n ' hca_port_pairs+=("${hca}:1")\n'\n ' done\n'\n '\n'\n ' active_hca_list=$(IFS=,; echo '\n '"${active_hcas[*]}")\n'\n ' hca_port_pairs_list=$(IFS=,; echo '\n '"${hca_port_pairs[*]}")\n'\n ' echo "[Infer RoCE] Setting active '\n 'HCAs: ${active_hca_list}"\n'\n ' export '\n 'NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n'\n ' export '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n'\n ' export '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'NCCL_IB_HCA=${NCCL_IB_HCA}"\n'\n ' echo "[Infer RoCE] '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n'\n ' echo "[Infer RoCE] '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n'\n ' else\n'\n ' echo "[Infer RoCE] WARNING: No '\n 'active RoCE HCAs found. NCCL_IB_HCA '\n 'will not be set."\n'\n ' fi\n'\n '\n'\n ' if [ ${#active_hcas[@]} -gt 0 ]; '\n 'then\n'\n ' echo "[Infer RoCE] Finding '\n 'GID_INDEX for each active HCA (SR-IOV '\n 'compatible)..."\n'\n '\n'\n ' # For SR-IOV environments, find '\n 'the most common IPv4 RoCE v2 GID index '\n 'across all HCAs\n'\n ' declare -A gid_index_count\n'\n ' declare -A [e2e-llm-inference-service] hca_gid_index\n'\n '\n'\n ' for hca_name in '\n '"${active_hcas[@]}"; do\n'\n ' echo "[Infer RoCE] Processing '\n 'HCA: ${hca_name}"\n'\n '\n'\n ' # Find all RoCE v2 IPv4 GIDs '\n 'for this HCA and count by index\n'\n ' for tpath in '\n '/sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; '\n 'do\n'\n ' if grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '"$tpath" 2>/dev/null; then\n'\n ' idx=$(basename '\n '"$tpath")\n'\n ' '\n 'gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n'\n ' # Check for IPv4 GID '\n '(contains ffff:)\n'\n ' if [ -f "$gid_file" ] '\n '&& grep -q "ffff:" "$gid_file"; then\n'\n ' gid_value=$(cat '\n '"$gid_file" 2>/dev/null || echo "")\n'\n ' echo "[Infer '\n 'RoCE] Found IPv4 RoCE v2 GID for '\n '${hca_name}: index=${idx}, '\n 'gid=${gid_value}"\n'\n ' '\n 'hca_gid_index["${hca_name}"]="${idx}"\n'\n ' '\n 'gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} '\n '+ 1))\n'\n ' break # Use '\n 'first found IPv4 GID per HCA\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n ' done\n'\n '\n'\n ' # Find the most common GID index '\n '(most likely to be consistent across '\n 'nodes)\n'\n ' best_gid_index=""\n'\n ' max_count=0\n'\n ' for idx in '\n '"${!gid_index_count[@]}"; do\n'\n ' '\n 'count=${gid_index_count["${idx}"]}\n'\n ' echo "[Infer RoCE] GID_INDEX '\n '${idx} found on ${count} HCAs"\n'\n ' if [ $count -gt $max_count ]; '\n 'then\n'\n ' max_count=$count\n'\n ' best_gid_index="$idx"\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Use deterministic fallback if '\n [e2e-llm-inference-service] 'tied - prefer index 3 (SR-IOV '\n 'standard)\n'\n ' if [ ${#gid_index_count[@]} -gt 1 '\n ']; then\n'\n ' echo "[Infer RoCE] Multiple '\n 'GID indices found, selecting most '\n 'common: ${best_gid_index}"\n'\n " # If there's a tie, prefer "\n "index 3 as it's most common in SR-IOV "\n 'setups\n'\n ' if [ -n '\n '"${gid_index_count[\'3\']}" ] && [ '\n '"${gid_index_count[\'3\']}" -eq '\n '"$max_count" ]; then\n'\n ' best_gid_index="3"\n'\n ' echo "[Infer RoCE] Using '\n 'deterministic fallback: GID_INDEX=3 '\n '(SR-IOV standard)"\n'\n ' fi\n'\n ' fi\n'\n '\n'\n ' # Check if GID_INDEX is already '\n 'set via environment variables\n'\n ' if [ -n "${NCCL_IB_GID_INDEX}" ]; '\n 'then\n'\n ' echo "[Infer RoCE] Using '\n 'pre-configured '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} '\n 'from environment"\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' echo "[Infer RoCE] Using '\n 'pre-configured '\n 'GID_INDEX=${NCCL_IB_GID_INDEX} for '\n 'NCCL, NVSHMEM, and UCX"\n'\n ' elif [ -n "$best_gid_index" ]; '\n 'then\n'\n ' echo "[Infer RoCE] Selected '\n 'GID_INDEX: ${best_gid_index} (found on '\n '${max_count} HCAs)"\n'\n '\n'\n ' export '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n'\n '\n'\n ' echo "[Infer RoCE] Exported '\n 'GID_INDEX=${best_gid_index} for NCCL, '\n 'NVSHMEM, and UCX"\n'\n ' else\n'\n ' echo "[Infer RoCE] ERROR: No '\n 'valid IPv4 '\n '${KSERVE_INFER_IB_GID_INDEX_GREP} '\n 'GID_INDEX found on any HCA."\n'\n ' fi\n'\n [e2e-llm-inference-service] ' else\n'\n ' echo "[Infer RoCE] No active HCAs '\n 'found, skipping GID_INDEX inference."\n'\n ' fi\n'\n 'fi\n'\n '\n'\n '# --disable-access-log-for-endpoints '\n 'landed in vLLM 0.16.0 '\n '(vllm-project/vllm#30011).\n'\n '# Older versions still need the blanket '\n '--disable-uvicorn-access-log.\n'\n 'ACCESS_LOG_ARGS="--disable-uvicorn-access-log"\n'\n 'VLLM_VERSION=$(vllm --version '\n "2>/dev/null | tail -1 | awk '{print "\n "$NF}')\n"\n 'echo "[access-log-detect] vllm '\n 'version=\'${VLLM_VERSION}\'"\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ "$(printf '\n '\'%s\\n%s\\n\' "0.16.0" '\n '"${VLLM_VERSION}" | sort -V | head -1)" '\n '= "0.16.0" ]; then\n'\n ' '\n 'ACCESS_LOG_ARGS="--disable-access-log-for-endpoints '\n '/health,/metrics,/ping"\n'\n 'fi\n'\n 'echo "[access-log-detect] selected '\n 'ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n'\n '\n'\n '# --shutdown-timeout landed in vLLM '\n '0.18.0 (vllm-project/vllm#36666).\n'\n 'SHUTDOWN_TIMEOUT_ARGS=""\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ "$(printf '\n '\'%s\\n%s\\n\' "0.18.0" '\n '"${VLLM_VERSION}" | sort -V | head -1)" '\n '= "0.18.0" ]; then\n'\n ' '\n 'SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout '\n '40"\n'\n 'fi\n'\n '\n'\n 'eval "exec vllm serve /mnt/models \\\n'\n ' --served-model-name '\n '"facebook/opt-125m" '\n '"publishers/kserve-ci-e2e-test/models/facebook/opt-125m" '\n '\\\n'\n ' --port 8000 \\\n'\n ' ${ACCESS_LOG_ARGS} \\\n'\n ' ${SHUTDOWN_TIMEOUT_ARGS} \\\n'\n ' --enable-ssl-refresh \\\n'\n ' --ssl-certfile '\n '/var/run/kserve/tls/tls.crt \\\n'\n ' --ssl-keyfile '\n '/var/run/kserve/tls/tls.key \\\n'\n ' ${VLLM_ADDITIONAL_ARGS} \\\n'\n ' $@"',\n '--'],\n 'env': [{'name': 'HOME',\n 'value': '/home',\n 'value_from': None},\n {'name': 'VLLM_LOGGING_LEVEL',\n [e2e-llm-inference-service] 'value': 'DEBUG',\n 'value_from': None},\n {'name': 'VLLM_CPU_KVCACHE_SPACE',\n 'value': '1',\n 'value_from': None},\n {'name': 'VLLM_ENABLE_V1_MULTIPROCESSING',\n 'value': '0',\n 'value_from': None},\n {'name': 'USER',\n 'value': 'nonroot',\n 'value_from': None},\n {'name': 'TORCHINDUCTOR_CACHE_DIR',\n 'value': '/tmp/torchinductor-cache',\n 'value_from': None},\n {'name': 'HF_HUB_CACHE',\n 'value': '/models',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': {'post_start': None,\n 'pre_stop': {'_exec': {'command': ['/bin/sleep',\n '15']},\n 'http_get': None,\n 'sleep': None,\n 'tcp_socket': None}},\n 'liveness_probe': {'_exec': None,\n 'failure_threshold': 10,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'name': 'main',\n 'ports': [{'container_port': 8000,\n 'host_ip': None,\n 'host_port': None,\n 'name': None,\n 'protocol': 'TCP'}],\n 'readiness_probe': {'_exec': None,\n 'failure_threshold': 2,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 1,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'resize_policy': None,\n 'resources': {'clai [e2e-llm-inference-service] ms': None,\n 'limits': {'cpu': '2', 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n 'read_only_root_filesystem': True,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'windows_options': None},\n 'startup_probe': {'_exec': None,\n 'failure_threshold': 60,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/home',\n 'mount_propagation': None,\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/tmp',\n 'mount_propagation': None,\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/dev/shm',\n 'mount_propagation': None,\n 'name': 'dshm',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n [e2e-llm-inference-service] 'sub_path_expr': None},\n {'mount_path': '/models',\n 'mount_propagation': None,\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/kserve/tls',\n 'mount_propagation': None,\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'dns_config': None,\n 'dns_policy': 'ClusterFirst',\n 'enable_service_links': True,\n 'ephemeral_containers': None,\n 'host_aliases': None,\n 'host_ipc': None,\n 'host_network': None,\n 'host_pid': None,\n 'host_users': None,\n 'hostname': None,\n 'image_pull_secrets': [{'name': 'default-dockercfg-6qcsh'}],\n 'init_containers': [{'args': ['hf://facebook/opt-125m',\n '/mnt/models',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-1',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-2'],\n 'command': None,\n 'env': [{'name': 'AWS_ACCESS_KEY_ID',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': None,\n 'secret_key_ref': {'key': 'AWS_ACCESS_KEY_ID',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'AWS_SECRET_ACCESS_KEY',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': None,\n [e2e-llm-inference-service] 'secret_key_ref': {'key': 'AWS_SECRET_ACCESS_KEY',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'S3_USE_HTTPS',\n 'value': '0',\n 'value_from': None},\n {'name': 'S3_ENDPOINT',\n 'value': 's3-service.kserve:8333',\n 'value_from': None},\n {'name': 'AWS_ENDPOINT_URL',\n 'value': 'http://s3-service.kserve:8333',\n 'value_from': None},\n {'name': 'S3_VERIFY_SSL',\n 'value': '0',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE',\n 'value': '/etc/ssl/custom-certs/cabundle.crt',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE_CONFIGMAP',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'HF_HUB_ENABLE_HF_TRANSFER',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_HIGH_PERFORMANCE',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_NUM_CONCURRENT_RANGE_GETS',\n 'value': '8',\n 'value_from': None},\n {'name': 'CA_BUNDLE_CONFIGMAP_NAME',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'CA_BUNDLE_VOLUME_MOUNT_POINT',\n 'value': '/etc/ssl/custom-certs',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': None,\n 'liveness_probe': None,\n 'name': 'storage-initializer',\n 'ports': None,\n 'readiness_probe': None,\n 'resize_policy': None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n 'read_only_root_filesy [e2e-llm-inference-service] stem': None,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': None,\n 'windows_options': None},\n 'startup_probe': None,\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'mount_propagation': None,\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'node_name': 'ip-10-0-131-57.ec2.internal',\n 'node_selector': None,\n 'os': None,\n 'overhead': None,\n 'preemption_policy': 'PreemptLowerPriority',\n 'priority': 0,\n 'priority_class_name': None,\n 'readiness_gates': None,\n 'resource_claims': None,\n 'resources': None,\n 'restart_policy': 'Always',\n 'runtime_class_name': None,\n 'scheduler_name': 'default-scheduler',\n 'scheduling_gates': None,\n 'security_context': {'app_armor_profile': None,\n 'fs_group': 1000690000,\n 'fs_group_change_policy': None,\n 'run_as_group': None,\n 'run_as_non_root': None,\n 'run_as_user': None,\n 'se_linux_change_policy': None,\n 'se_linux_options': {'level': 's0:c26,c20',\n 'role': None,\n 'type': None,\n 'user': None},\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'supplemental_groups': None,\n 'supplemental_groups_poli [e2e-llm-inference-service] cy': None,\n 'sysctls': None,\n 'windows_options': None},\n 'service_account': 'default',\n 'service_account_name': 'default',\n 'set_hostname_as_fqdn': None,\n 'share_process_namespace': None,\n 'subdomain': None,\n 'termination_grace_period_seconds': 60,\n 'tolerations': [{'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/not-ready',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/unreachable',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoSchedule',\n 'key': 'node.kubernetes.io/memory-pressure',\n 'operator': 'Exists',\n 'toleration_seconds': None,\n 'value': None}],\n 'topology_spread_constraints': None,\n 'volumes': [{'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'home',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': 'Memory', 'size_limit': '1Gi'},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'dshm',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_di [e2e-llm-inference-service] sk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'model-cache',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tmp-dir',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tls-certs',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': {'default_mode': 420,\n 'items': None,\n 'optional': None,\n [e2e-llm-inference-service] 'secret_name': 'lora-multiple-adapters-test-kserve-self-signed-certs'},\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kserve-provision-location',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': {'default_mode': 420,\n 'items': None,\n 'name': 'odh-kserve-custom-ca-bundle',\n 'optional': None},\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'cabundle-cert',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kube-api-access-n5dws',\n 'nfs': None,\n 'persis [e2e-llm-inference-service] tent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': {'default_mode': 420,\n 'sources': [{'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': {'audience': None,\n 'expiration_seconds': 3607,\n 'path': 'token'}},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'ca.crt',\n 'mode': None,\n 'path': 'ca.crt'}],\n 'name': 'kube-root-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': {'items': [{'field_ref': {'api_version': 'v1',\n 'field_path': 'metadata.namespace'},\n 'mode': None,\n 'path': 'namespace',\n 'resource_field_ref': None}]},\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'service-ca.crt',\n 'mode': None,\n 'path': 'service-ca.crt'}],\n 'name': 'openshift-service-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None}]},\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None}]},\n 'status': {'conditions': [{'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 18, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodReadyToStartContainers'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n [e2e-llm-inference-service] 'status': 'True',\n 'type': 'Initialized'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'Ready'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'ContainersReady'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodScheduled'}],\n 'container_statuses': [{'allocated_resources': {'cpu': '200m',\n 'memory': '2Gi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402',\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_id': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'main',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2',\n 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_count': 0,\n 'started': True,\n 'state': {'running': {'started_at': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal())},\n 'terminated': None,\n 'waiting': None},\n 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/home',\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/tmp',\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/dev/shm',\n 'name': 'dshm',\n 'read_only': None,\n 'recursiv [e2e-llm-inference-service] e_read_only': None},\n {'mount_path': '/models',\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/var/run/kserve/tls',\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'ephemeral_container_statuses': None,\n 'host_i_ps': [{'ip': '10.0.131.57'}],\n 'host_ip': '10.0.131.57',\n 'init_container_statuses': [{'allocated_resources': {'cpu': '100m',\n 'memory': '100Mi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_id': 'quay.io/opendatahub/kserve-storage-initializer@sha256:60e2a5c8616d82c8933b614ae003dd3412a420a29a0be570b6a6439a3d39ab99',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'storage-initializer',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_count': 0,\n 'started': False,\n 'state': {'running': None,\n 'terminated': {'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'exit_code': 0,\n 'finished_at': datetime.datetime(2026, 7, 6, 21, 35, 22, tzinfo=tzlocal()),\n 'message': None,\n 'reason': 'Completed',\n 'signal': None,\n 'started_at': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n 'waiting': None},\n [e2e-llm-inference-service] 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'message': None,\n 'nominated_node_name': None,\n 'phase': 'Running',\n 'pod_i_ps': [{'ip': '10.134.0.76'}],\n 'pod_ip': '10.134.0.76',\n 'qos_class': 'Burstable',\n 'reason': None,\n 'resize': None,\n 'resource_claim_statuses': None,\n 'start_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())}}] [e2e-llm-inference-service] + where [{'api_version': None,\n 'kind': None,\n 'metadata': {'annotations': {'k8s.ovn.org/pod-networks': '{"default":{"ip_addresses":["10.134.0.76/23"],"mac_address":"0a:58:0a:86:00:4c","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.76/23","gateway_ip":"10.134.0.1","role":"primary"}}',\n 'k8s.v1.cni.cncf.io/network-status': '[{\n'\n ' '\n '"name": '\n '"ovn-kubernetes",\n'\n ' '\n '"interface": '\n '"eth0",\n'\n ' "ips": '\n '[\n'\n ' '\n '"10.134.0.76"\n'\n ' ],\n'\n ' "mac": '\n '"0a:58:0a:86:00:4c",\n'\n ' '\n '"default": '\n 'true,\n'\n ' "dns": '\n '{}\n'\n '}]',\n 'openshift.io/scc': 'restricted-v2',\n 'seccomp.security.alpha.kubernetes.io/pod': 'runtime/default',\n 'security.openshift.io/validated-scc-subject-type': 'user'},\n 'creation_timestamp': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'deletion_grace_period_seconds': None,\n 'deletion_timestamp': None,\n 'finalizers': None,\n 'generate_name': 'lora-multiple-adapters-test-kserve-77485bff95-',\n 'generation': 1,\n 'labels': {'app.kubernetes.io/component': 'llminferenceservice-workload',\n 'app.kubernetes.io/name': 'lora-multiple-adapters-test',\n 'app.kubernetes.io/part-of': 'llminferenceservice',\n 'kserve.io/component': 'workload',\n 'llm-d.ai/role': 'both',\n 'pod-template-hash': '77485bff95'},\n 'managed_fields': [{'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.ovn.org/pod-networks': {}}}},\n 'manager': 'ip-10-0-131-57',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:generateName': {},\n 'f:labels': {'.': {},\n 'f:app.kubernetes.io/component': {},\n [e2e-llm-inference-service] 'f:app.kubernetes.io/name': {},\n 'f:app.kubernetes.io/part-of': {},\n 'f:kserve.io/component': {},\n 'f:llm-d.ai/role': {},\n 'f:pod-template-hash': {}},\n 'f:ownerReferences': {'.': {},\n 'k:{"uid":"75c381c4-1742-4fb9-b05f-92ea6db35c9c"}': {}}},\n 'f:spec': {'f:containers': {'k:{"name":"main"}': {'.': {},\n 'f:args': {},\n 'f:command': {},\n 'f:env': {'.': {},\n 'k:{"name":"HF_HUB_CACHE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HOME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"TORCHINDUCTOR_CACHE_DIR"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"USER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_CPU_KVCACHE_SPACE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{ [e2e-llm-inference-service] "name":"VLLM_LOGGING_LEVEL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:lifecycle': {'.': {},\n 'f:preStop': {'.': {},\n 'f:exec': {'.': {},\n 'f:command': {}}}},\n 'f:livenessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:name': {},\n 'f:ports': {'.': {},\n 'k:{"containerPort":8000,"protocol":"TCP"}': {'.': {},\n 'f:containerPort': {},\n 'f:protocol': {}}},\n 'f:readinessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n [e2e-llm-inference-service] 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:securityContext': {'.': {},\n 'f:allowPrivilegeEscalation': {},\n 'f:capabilities': {'.': {},\n 'f:drop': {}},\n 'f:readOnlyRootFilesystem': {},\n 'f:runAsNonRoot': {},\n 'f:seccompProfile': {'.': {},\n 'f:type': {}}},\n 'f:startupProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:terminationMessagePath': {},\n [e2e-llm-inference-service] 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/dev/shm"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/home"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/models"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/tmp"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/var/run/kserve/tls"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}}}}},\n 'f:dnsPolicy': {},\n 'f:enableServiceLinks': {},\n 'f:initContainers': {'.': {},\n 'k:{"name":"storage-initializer"}': {'.': {},\n 'f:args': {},\n 'f:env': {'.': [e2e-llm-inference-service] {},\n 'k:{"name":"AWS_ACCESS_KEY_ID"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"AWS_CA_BUNDLE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_ENDPOINT_URL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_SECRET_ACCESS_KEY"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}': {'.': {},\n 'f:name': {},\n [e2e-llm-inference-service] 'f:value': {}},\n 'k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_HIGH_PERFORMANCE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_ENDPOINT"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_USE_HTTPS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_VERIFY_SSL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:name': {},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n [e2e-llm-inference-service] 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:terminationMessagePath': {},\n 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/etc/ssl/custom-certs"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}}}}},\n 'f:restartPolicy': {},\n 'f:schedulerName': {},\n 'f:securityContext': {},\n 'f:terminationGracePeriodSeconds': {},\n 'f:volumes': {'.': {},\n 'k:{"name":"cabundle-cert"}': {'.': {},\n 'f:configMap': {'.': {},\n 'f:defaultMode': {},\n 'f:name': {}},\n 'f:name': {}},\n 'k:{"name":"dshm"}': {'.': {},\n 'f:emptyDir': {'.': {},\n 'f:medium': {},\n 'f:sizeLimit': {}},\n [e2e-llm-inference-service] 'f:name': {}},\n 'k:{"name":"home"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"kserve-provision-location"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"model-cache"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"tls-certs"}': {'.': {},\n 'f:name': {},\n 'f:secret': {'.': {},\n 'f:defaultMode': {},\n 'f:secretName': {}}},\n 'k:{"name":"tmp-dir"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}}}}},\n 'manager': 'kube-controller-manager',\n 'operation': 'Update',\n 'subresource': None,\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.v1.cni.cncf.io/network-status': {}}}},\n 'manager': 'multus-daemon',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:status': {'f:conditions': {'k:{"type":"ContainersReady"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"Initialized"}': {'.': {},\n 'f:lastProbeTime': {},\n [e2e-llm-inference-service] 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodReadyToStartContainers"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodScheduled"}': {'f:observedGeneration': {}},\n 'k:{"type":"Ready"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}}},\n 'f:containerStatuses': {},\n 'f:hostIP': {},\n 'f:hostIPs': {},\n 'f:initContainerStatuses': {},\n 'f:observedGeneration': {},\n 'f:phase': {},\n 'f:podIP': {},\n 'f:podIPs': {'.': {},\n 'k:{"ip":"10.134.0.76"}': {'.': {},\n 'f:ip': {}}},\n 'f:startTime': {}}},\n 'manager': 'kubelet',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal())}],\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95-2tx5t',\n 'namespace': 'kserve-ci-e2e-test',\n 'owner_references': [{'api_version': 'apps/v1',\n 'block_owner_deletion': True,\n 'controller': True,\n 'kind': 'ReplicaSet',\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95',\n 'uid': '75c381c4-1742-4fb9-b05f-92ea6db35c9c'}],\n 'resource_version': '64798',\n [e2e-llm-inference-service] 'self_link': None,\n 'uid': 'd4948f86-f0d1-4911-908f-7cb88db94468'},\n 'spec': {'active_deadline_seconds': None,\n 'affinity': None,\n 'automount_service_account_token': None,\n 'containers': [{'args': ['--enable-lora',\n '--max-lora-rank=64',\n '--max-loras=2',\n '--max-cpu-loras=4',\n '--lora-modules',\n '\'{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\''],\n 'command': ['/bin/bash',\n '-c',\n 'if [ -f /etc/profile.d/ibm-aiu-setup.sh '\n ']; then\n'\n ' source '\n '/etc/profile.d/ibm-aiu-setup.sh\n'\n 'fi\n'\n '\n'\n 'if [ "$KSERVE_INFER_ROCE" = "true" ]; '\n 'then\n'\n ' echo "Trying to infer RoCE configs '\n '... "\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gids/* '\n '2>/dev/null\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gid_attrs/types/* '\n '2>/dev/null\n'\n '\n'\n ' cat /proc/driver/nvidia/params\n'\n '\n'\n ' '\n 'KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE '\n 'v2"}\n'\n '\n'\n ' echo "[Infer RoCE] Discovering active '\n 'HCAs ..."\n'\n ' active_hcas=()\n'\n ' # Loop through all mlx5 devices found '\n 'in sysfs\n'\n ' for hca_dir in '\n '/sys/class/infiniband/mlx5_*; do\n'\n " # Ensure it's a directory before "\n 'proceeding\n'\n ' if [ -d "$hca_dir" ]; then\n'\n ' hca_name=$(basename '\n '"$hca_dir")\n'\n ' '\n 'port_state_file="$hca_dir/ports/1/state" '\n '# Assume port 1\n'\n ' '\n 'type_file="$hca_dir/ports/1/gid_attrs/types/*"\n'\n '\n'\n ' echo "[Infer RoCE] Check if '\n 'the port state file ${port_state_file} '\n 'exists and contains \'ACTIVE\'"\n'\n ' if [ -f "$port_state_file" ] '\n '&& grep -q "ACTIVE" "$port_state_file" '\n [e2e-llm-inference-service] '&& grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '${type_file} 2>/dev/null; then\n'\n ' echo "[Infer RoCE] Found '\n 'active HCA: $hca_name"\n'\n ' '\n 'active_hcas+=("$hca_name")\n'\n ' else\n'\n ' echo "[Infer RoCE] '\n 'Skipping inactive or down HCA: '\n '$hca_name"\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Check if we found any active HCAs\n'\n ' if [ ${#active_hcas[@]} -gt 0 ]; '\n 'then\n'\n ' # Join the array elements with a '\n 'comma\n'\n ' hca_port_pairs=()\n'\n ' for hca in "${active_hcas[@]}"; '\n 'do\n'\n ' hca_port_pairs+=("${hca}:1")\n'\n ' done\n'\n '\n'\n ' active_hca_list=$(IFS=,; echo '\n '"${active_hcas[*]}")\n'\n ' hca_port_pairs_list=$(IFS=,; echo '\n '"${hca_port_pairs[*]}")\n'\n ' echo "[Infer RoCE] Setting active '\n 'HCAs: ${active_hca_list}"\n'\n ' export '\n 'NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n'\n ' export '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n'\n ' export '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'NCCL_IB_HCA=${NCCL_IB_HCA}"\n'\n ' echo "[Infer RoCE] '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n'\n ' echo "[Infer RoCE] '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n'\n ' else\n'\n ' echo "[Infer RoCE] WARNING: No '\n 'active RoCE HCAs found. NCCL_IB_HCA '\n 'will not be set."\n'\n ' fi\n'\n '\n'\n ' if [ ${#active_hcas[@]} -gt 0 ]; '\n 'then\n'\n ' echo "[Infer RoCE] Finding '\n 'GID_INDEX for each active HCA (SR-IOV '\n 'compatible)..."\n'\n '\n'\n ' # For SR-IOV environments, find '\n 'the most common IPv4 RoCE v2 GID index '\n 'across all HCAs\n'\n ' declare -A gid_index_count\n'\n ' declare -A h [e2e-llm-inference-service] ca_gid_index\n'\n '\n'\n ' for hca_name in '\n '"${active_hcas[@]}"; do\n'\n ' echo "[Infer RoCE] Processing '\n 'HCA: ${hca_name}"\n'\n '\n'\n ' # Find all RoCE v2 IPv4 GIDs '\n 'for this HCA and count by index\n'\n ' for tpath in '\n '/sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; '\n 'do\n'\n ' if grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '"$tpath" 2>/dev/null; then\n'\n ' idx=$(basename '\n '"$tpath")\n'\n ' '\n 'gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n'\n ' # Check for IPv4 GID '\n '(contains ffff:)\n'\n ' if [ -f "$gid_file" ] '\n '&& grep -q "ffff:" "$gid_file"; then\n'\n ' gid_value=$(cat '\n '"$gid_file" 2>/dev/null || echo "")\n'\n ' echo "[Infer '\n 'RoCE] Found IPv4 RoCE v2 GID for '\n '${hca_name}: index=${idx}, '\n 'gid=${gid_value}"\n'\n ' '\n 'hca_gid_index["${hca_name}"]="${idx}"\n'\n ' '\n 'gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} '\n '+ 1))\n'\n ' break # Use '\n 'first found IPv4 GID per HCA\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n ' done\n'\n '\n'\n ' # Find the most common GID index '\n '(most likely to be consistent across '\n 'nodes)\n'\n ' best_gid_index=""\n'\n ' max_count=0\n'\n ' for idx in '\n '"${!gid_index_count[@]}"; do\n'\n ' '\n 'count=${gid_index_count["${idx}"]}\n'\n ' echo "[Infer RoCE] GID_INDEX '\n '${idx} found on ${count} HCAs"\n'\n ' if [ $count -gt $max_count ]; '\n 'then\n'\n ' max_count=$count\n'\n ' best_gid_index="$idx"\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Use deterministic fallback if '\n [e2e-llm-inference-service] 'tied - prefer index 3 (SR-IOV '\n 'standard)\n'\n ' if [ ${#gid_index_count[@]} -gt 1 '\n ']; then\n'\n ' echo "[Infer RoCE] Multiple '\n 'GID indices found, selecting most '\n 'common: ${best_gid_index}"\n'\n " # If there's a tie, prefer "\n "index 3 as it's most common in SR-IOV "\n 'setups\n'\n ' if [ -n '\n '"${gid_index_count[\'3\']}" ] && [ '\n '"${gid_index_count[\'3\']}" -eq '\n '"$max_count" ]; then\n'\n ' best_gid_index="3"\n'\n ' echo "[Infer RoCE] Using '\n 'deterministic fallback: GID_INDEX=3 '\n '(SR-IOV standard)"\n'\n ' fi\n'\n ' fi\n'\n '\n'\n ' # Check if GID_INDEX is already '\n 'set via environment variables\n'\n ' if [ -n "${NCCL_IB_GID_INDEX}" ]; '\n 'then\n'\n ' echo "[Infer RoCE] Using '\n 'pre-configured '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} '\n 'from environment"\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' echo "[Infer RoCE] Using '\n 'pre-configured '\n 'GID_INDEX=${NCCL_IB_GID_INDEX} for '\n 'NCCL, NVSHMEM, and UCX"\n'\n ' elif [ -n "$best_gid_index" ]; '\n 'then\n'\n ' echo "[Infer RoCE] Selected '\n 'GID_INDEX: ${best_gid_index} (found on '\n '${max_count} HCAs)"\n'\n '\n'\n ' export '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n'\n '\n'\n ' echo "[Infer RoCE] Exported '\n 'GID_INDEX=${best_gid_index} for NCCL, '\n 'NVSHMEM, and UCX"\n'\n ' else\n'\n ' echo "[Infer RoCE] ERROR: No '\n 'valid IPv4 '\n '${KSERVE_INFER_IB_GID_INDEX_GREP} '\n 'GID_INDEX found on any HCA."\n'\n ' fi\n'\n [e2e-llm-inference-service] ' else\n'\n ' echo "[Infer RoCE] No active HCAs '\n 'found, skipping GID_INDEX inference."\n'\n ' fi\n'\n 'fi\n'\n '\n'\n '# --disable-access-log-for-endpoints '\n 'landed in vLLM 0.16.0 '\n '(vllm-project/vllm#30011).\n'\n '# Older versions still need the blanket '\n '--disable-uvicorn-access-log.\n'\n 'ACCESS_LOG_ARGS="--disable-uvicorn-access-log"\n'\n 'VLLM_VERSION=$(vllm --version '\n "2>/dev/null | tail -1 | awk '{print "\n "$NF}')\n"\n 'echo "[access-log-detect] vllm '\n 'version=\'${VLLM_VERSION}\'"\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ "$(printf '\n '\'%s\\n%s\\n\' "0.16.0" '\n '"${VLLM_VERSION}" | sort -V | head -1)" '\n '= "0.16.0" ]; then\n'\n ' '\n 'ACCESS_LOG_ARGS="--disable-access-log-for-endpoints '\n '/health,/metrics,/ping"\n'\n 'fi\n'\n 'echo "[access-log-detect] selected '\n 'ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n'\n '\n'\n '# --shutdown-timeout landed in vLLM '\n '0.18.0 (vllm-project/vllm#36666).\n'\n 'SHUTDOWN_TIMEOUT_ARGS=""\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ "$(printf '\n '\'%s\\n%s\\n\' "0.18.0" '\n '"${VLLM_VERSION}" | sort -V | head -1)" '\n '= "0.18.0" ]; then\n'\n ' '\n 'SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout '\n '40"\n'\n 'fi\n'\n '\n'\n 'eval "exec vllm serve /mnt/models \\\n'\n ' --served-model-name '\n '"facebook/opt-125m" '\n '"publishers/kserve-ci-e2e-test/models/facebook/opt-125m" '\n '\\\n'\n ' --port 8000 \\\n'\n ' ${ACCESS_LOG_ARGS} \\\n'\n ' ${SHUTDOWN_TIMEOUT_ARGS} \\\n'\n ' --enable-ssl-refresh \\\n'\n ' --ssl-certfile '\n '/var/run/kserve/tls/tls.crt \\\n'\n ' --ssl-keyfile '\n '/var/run/kserve/tls/tls.key \\\n'\n ' ${VLLM_ADDITIONAL_ARGS} \\\n'\n ' $@"',\n '--'],\n 'env': [{'name': 'HOME',\n 'value': '/home',\n 'value_from': None},\n {'name': 'VLLM_LOGGING_LEVEL',\n [e2e-llm-inference-service] 'value': 'DEBUG',\n 'value_from': None},\n {'name': 'VLLM_CPU_KVCACHE_SPACE',\n 'value': '1',\n 'value_from': None},\n {'name': 'VLLM_ENABLE_V1_MULTIPROCESSING',\n 'value': '0',\n 'value_from': None},\n {'name': 'USER',\n 'value': 'nonroot',\n 'value_from': None},\n {'name': 'TORCHINDUCTOR_CACHE_DIR',\n 'value': '/tmp/torchinductor-cache',\n 'value_from': None},\n {'name': 'HF_HUB_CACHE',\n 'value': '/models',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': {'post_start': None,\n 'pre_stop': {'_exec': {'command': ['/bin/sleep',\n '15']},\n 'http_get': None,\n 'sleep': None,\n 'tcp_socket': None}},\n 'liveness_probe': {'_exec': None,\n 'failure_threshold': 10,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'name': 'main',\n 'ports': [{'container_port': 8000,\n 'host_ip': None,\n 'host_port': None,\n 'name': None,\n 'protocol': 'TCP'}],\n 'readiness_probe': {'_exec': None,\n 'failure_threshold': 2,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 1,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'resize_policy': None,\n 'resources': {'claim [e2e-llm-inference-service] s': None,\n 'limits': {'cpu': '2', 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n 'read_only_root_filesystem': True,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'windows_options': None},\n 'startup_probe': {'_exec': None,\n 'failure_threshold': 60,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/home',\n 'mount_propagation': None,\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/tmp',\n 'mount_propagation': None,\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/dev/shm',\n 'mount_propagation': None,\n 'name': 'dshm',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n [e2e-llm-inference-service] 'sub_path_expr': None},\n {'mount_path': '/models',\n 'mount_propagation': None,\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/kserve/tls',\n 'mount_propagation': None,\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'dns_config': None,\n 'dns_policy': 'ClusterFirst',\n 'enable_service_links': True,\n 'ephemeral_containers': None,\n 'host_aliases': None,\n 'host_ipc': None,\n 'host_network': None,\n 'host_pid': None,\n 'host_users': None,\n 'hostname': None,\n 'image_pull_secrets': [{'name': 'default-dockercfg-6qcsh'}],\n 'init_containers': [{'args': ['hf://facebook/opt-125m',\n '/mnt/models',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-1',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-2'],\n 'command': None,\n 'env': [{'name': 'AWS_ACCESS_KEY_ID',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': None,\n 'secret_key_ref': {'key': 'AWS_ACCESS_KEY_ID',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'AWS_SECRET_ACCESS_KEY',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': None,\n [e2e-llm-inference-service] 'secret_key_ref': {'key': 'AWS_SECRET_ACCESS_KEY',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'S3_USE_HTTPS',\n 'value': '0',\n 'value_from': None},\n {'name': 'S3_ENDPOINT',\n 'value': 's3-service.kserve:8333',\n 'value_from': None},\n {'name': 'AWS_ENDPOINT_URL',\n 'value': 'http://s3-service.kserve:8333',\n 'value_from': None},\n {'name': 'S3_VERIFY_SSL',\n 'value': '0',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE',\n 'value': '/etc/ssl/custom-certs/cabundle.crt',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE_CONFIGMAP',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'HF_HUB_ENABLE_HF_TRANSFER',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_HIGH_PERFORMANCE',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_NUM_CONCURRENT_RANGE_GETS',\n 'value': '8',\n 'value_from': None},\n {'name': 'CA_BUNDLE_CONFIGMAP_NAME',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'CA_BUNDLE_VOLUME_MOUNT_POINT',\n 'value': '/etc/ssl/custom-certs',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': None,\n 'liveness_probe': None,\n 'name': 'storage-initializer',\n 'ports': None,\n 'readiness_probe': None,\n 'resize_policy': None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n 'read_only_root_filesys [e2e-llm-inference-service] tem': None,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': None,\n 'windows_options': None},\n 'startup_probe': None,\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'mount_propagation': None,\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'node_name': 'ip-10-0-131-57.ec2.internal',\n 'node_selector': None,\n 'os': None,\n 'overhead': None,\n 'preemption_policy': 'PreemptLowerPriority',\n 'priority': 0,\n 'priority_class_name': None,\n 'readiness_gates': None,\n 'resource_claims': None,\n 'resources': None,\n 'restart_policy': 'Always',\n 'runtime_class_name': None,\n 'scheduler_name': 'default-scheduler',\n 'scheduling_gates': None,\n 'security_context': {'app_armor_profile': None,\n 'fs_group': 1000690000,\n 'fs_group_change_policy': None,\n 'run_as_group': None,\n 'run_as_non_root': None,\n 'run_as_user': None,\n 'se_linux_change_policy': None,\n 'se_linux_options': {'level': 's0:c26,c20',\n 'role': None,\n 'type': None,\n 'user': None},\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'supplemental_groups': None,\n 'supplemental_groups_polic [e2e-llm-inference-service] y': None,\n 'sysctls': None,\n 'windows_options': None},\n 'service_account': 'default',\n 'service_account_name': 'default',\n 'set_hostname_as_fqdn': None,\n 'share_process_namespace': None,\n 'subdomain': None,\n 'termination_grace_period_seconds': 60,\n 'tolerations': [{'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/not-ready',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/unreachable',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoSchedule',\n 'key': 'node.kubernetes.io/memory-pressure',\n 'operator': 'Exists',\n 'toleration_seconds': None,\n 'value': None}],\n 'topology_spread_constraints': None,\n 'volumes': [{'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'home',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': 'Memory', 'size_limit': '1Gi'},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'dshm',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_dis [e2e-llm-inference-service] k': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'model-cache',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tmp-dir',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tls-certs',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': {'default_mode': 420,\n 'items': None,\n 'optional': None,\n [e2e-llm-inference-service] 'secret_name': 'lora-multiple-adapters-test-kserve-self-signed-certs'},\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kserve-provision-location',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': {'default_mode': 420,\n 'items': None,\n 'name': 'odh-kserve-custom-ca-bundle',\n 'optional': None},\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'cabundle-cert',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kube-api-access-n5dws',\n 'nfs': None,\n 'persist [e2e-llm-inference-service] ent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': {'default_mode': 420,\n 'sources': [{'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': {'audience': None,\n 'expiration_seconds': 3607,\n 'path': 'token'}},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'ca.crt',\n 'mode': None,\n 'path': 'ca.crt'}],\n 'name': 'kube-root-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': {'items': [{'field_ref': {'api_version': 'v1',\n 'field_path': 'metadata.namespace'},\n 'mode': None,\n 'path': 'namespace',\n 'resource_field_ref': None}]},\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'service-ca.crt',\n 'mode': None,\n 'path': 'service-ca.crt'}],\n 'name': 'openshift-service-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None}]},\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None}]},\n 'status': {'conditions': [{'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 18, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodReadyToStartContainers'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n [e2e-llm-inference-service] 'status': 'True',\n 'type': 'Initialized'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'Ready'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'ContainersReady'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodScheduled'}],\n 'container_statuses': [{'allocated_resources': {'cpu': '200m',\n 'memory': '2Gi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402',\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_id': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'main',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2',\n 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_count': 0,\n 'started': True,\n 'state': {'running': {'started_at': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal())},\n 'terminated': None,\n 'waiting': None},\n 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/home',\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/tmp',\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/dev/shm',\n 'name': 'dshm',\n 'read_only': None,\n 'recursive [e2e-llm-inference-service] _read_only': None},\n {'mount_path': '/models',\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/var/run/kserve/tls',\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'ephemeral_container_statuses': None,\n 'host_i_ps': [{'ip': '10.0.131.57'}],\n 'host_ip': '10.0.131.57',\n 'init_container_statuses': [{'allocated_resources': {'cpu': '100m',\n 'memory': '100Mi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_id': 'quay.io/opendatahub/kserve-storage-initializer@sha256:60e2a5c8616d82c8933b614ae003dd3412a420a29a0be570b6a6439a3d39ab99',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'storage-initializer',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_count': 0,\n 'started': False,\n 'state': {'running': None,\n 'terminated': {'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'exit_code': 0,\n 'finished_at': datetime.datetime(2026, 7, 6, 21, 35, 22, tzinfo=tzlocal()),\n 'message': None,\n 'reason': 'Completed',\n 'signal': None,\n 'started_at': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n 'waiting': None},\n [e2e-llm-inference-service] 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'message': None,\n 'nominated_node_name': None,\n 'phase': 'Running',\n 'pod_i_ps': [{'ip': '10.134.0.76'}],\n 'pod_ip': '10.134.0.76',\n 'qos_class': 'Burstable',\n 'reason': None,\n 'resize': None,\n 'resource_claim_statuses': None,\n 'start_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())}}] = {'api_version': 'v1',\n 'items': [{'api_version': None,\n 'kind': None,\n 'metadata': {'annotations': {'k8s.ovn.org/pod-networks': '{"default":{"ip_addresses":["10.134.0.76/23"],"mac_address":"0a:58:0a:86:00:4c","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.76/23","gateway_ip":"10.134.0.1","role":"primary"}}',\n 'k8s.v1.cni.cncf.io/network-status': '[{\n'\n ' '\n '"name": '\n '"ovn-kubernetes",\n'\n ' '\n '"interface": '\n '"eth0",\n'\n ' '\n '"ips": '\n '[\n'\n ' '\n '"10.134.0.76"\n'\n ' '\n '],\n'\n ' '\n '"mac": '\n '"0a:58:0a:86:00:4c",\n'\n ' '\n [e2e-llm-inference-service] '"default": '\n 'true,\n'\n ' '\n '"dns": '\n '{}\n'\n '}]',\n 'openshift.io/scc': 'restricted-v2',\n 'seccomp.security.alpha.kubernetes.io/pod': 'runtime/default',\n 'security.openshift.io/validated-scc-subject-type': 'user'},\n 'creation_timestamp': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'deletion_grace_period_seconds': None,\n 'deletion_timestamp': None,\n 'finalizers': None,\n 'generate_name': 'lora-multiple-adapters-test-kserve-77485bff95-',\n 'generation': 1,\n 'labels': {'app.kubernetes.io/component': 'llminferenceservice-workload',\n 'app.kubernetes.io/name': 'lora-multiple-adapters-test',\n 'app.kubernetes.io/part-of': 'llminferenceservice',\n 'kserve.io/component': 'workload',\n 'llm-d.ai/role': 'both',\n 'pod-template-hash': '77485bff95'},\n 'managed_fields': [{'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.ovn.org/pod-networks': {}}}},\n 'manager': 'ip-10-0-131-57',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:generateName': {},\n 'f:labels': {'.': {},\n 'f:app.kubernetes.io/component': {},\n 'f:app.kubernetes.io/name': {},\n 'f:app.kubernetes.io/part-of': {},\n 'f:kserve.io/component': {},\n 'f:llm-d.ai/role': {},\n 'f:pod-template-hash': {}},\n 'f:ownerReferences': {'.': {},\n 'k:{"uid":"75c381c4-1742-4fb9-b05f-92ea6db35c9c"}': {}}},\n 'f:spec': {'f:containers': {'k:{"name":"main"}': {'.': {},\n 'f:args': {},\n 'f:command': {},\n 'f:env': {'.': {},\n [e2e-llm-inference-service] 'k:{"name":"HF_HUB_CACHE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HOME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"TORCHINDUCTOR_CACHE_DIR"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"USER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_CPU_KVCACHE_SPACE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_LOGGING_LEVEL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:lifecycle': {'.': {},\n 'f:preStop': {'.': {},\n [e2e-llm-inference-service] 'f:exec': {'.': {},\n 'f:command': {}}}},\n 'f:livenessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:name': {},\n 'f:ports': {'.': {},\n 'k:{"containerPort":8000,"protocol":"TCP"}': {'.': {},\n 'f:containerPort': {},\n 'f:protocol': {}}},\n 'f:readinessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:resources': {'.': {},\n [e2e-llm-inference-service] 'f:limits': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:securityContext': {'.': {},\n 'f:allowPrivilegeEscalation': {},\n 'f:capabilities': {'.': {},\n 'f:drop': {}},\n 'f:readOnlyRootFilesystem': {},\n 'f:runAsNonRoot': {},\n 'f:seccompProfile': {'.': {},\n 'f:type': {}}},\n 'f:startupProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:terminationMessagePath': {},\n 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/dev/shm"}': {'.': {},\n [e2e-llm-inference-service] 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/home"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/models"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/tmp"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/var/run/kserve/tls"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}}}}},\n 'f:dnsPolicy': {},\n 'f:enableServiceLinks': {},\n 'f:initContainers': {'.': {},\n 'k:{"name":"storage-initializer"}': {'.': {},\n 'f:args': {},\n 'f:env': {'.': {},\n [e2e-llm-inference-service] 'k:{"name":"AWS_ACCESS_KEY_ID"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"AWS_CA_BUNDLE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_ENDPOINT_URL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_SECRET_ACCESS_KEY"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}': {'.': {},\n [e2e-llm-inference-service] 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_HIGH_PERFORMANCE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_ENDPOINT"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_USE_HTTPS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_VERIFY_SSL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n [e2e-llm-inference-service] 'f:name': {},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:terminationMessagePath': {},\n 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/etc/ssl/custom-certs"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}}}}},\n 'f:restartPolicy': {},\n 'f:schedulerName': {},\n 'f:securityContext': {},\n 'f:terminationGracePeriodSeconds': {},\n 'f:volumes': {'.': {},\n 'k:{"name":"cabundle-cert"}': {'.': {},\n 'f:configMap': {'.': {},\n 'f:defaultMode': {},\n [e2e-llm-inference-service] 'f:name': {}},\n 'f:name': {}},\n 'k:{"name":"dshm"}': {'.': {},\n 'f:emptyDir': {'.': {},\n 'f:medium': {},\n 'f:sizeLimit': {}},\n 'f:name': {}},\n 'k:{"name":"home"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"kserve-provision-location"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"model-cache"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"tls-certs"}': {'.': {},\n 'f:name': {},\n 'f:secret': {'.': {},\n 'f:defaultMode': {},\n 'f:secretName': {}}},\n 'k:{"name":"tmp-dir"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}}}}},\n 'manager': 'kube-controller-manager',\n 'operation': 'Update',\n 'subresource': None,\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.v1.cni.cncf.io/network-status': {}}}},\n 'manager': 'multus-daemon',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n [e2e-llm-inference-service] 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:status': {'f:conditions': {'k:{"type":"ContainersReady"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"Initialized"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodReadyToStartContainers"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodScheduled"}': {'f:observedGeneration': {}},\n 'k:{"type":"Ready"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}}},\n 'f:containerStatuses': {},\n 'f:hostIP': {},\n 'f:hostIPs': {},\n 'f:initContainerStatuses': {},\n 'f:observedGeneration': {},\n [e2e-llm-inference-service] 'f:phase': {},\n 'f:podIP': {},\n 'f:podIPs': {'.': {},\n 'k:{"ip":"10.134.0.76"}': {'.': {},\n 'f:ip': {}}},\n 'f:startTime': {}}},\n 'manager': 'kubelet',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal())}],\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95-2tx5t',\n 'namespace': 'kserve-ci-e2e-test',\n 'owner_references': [{'api_version': 'apps/v1',\n 'block_owner_deletion': True,\n 'controller': True,\n 'kind': 'ReplicaSet',\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95',\n 'uid': '75c381c4-1742-4fb9-b05f-92ea6db35c9c'}],\n 'resource_version': '64798',\n 'self_link': None,\n 'uid': 'd4948f86-f0d1-4911-908f-7cb88db94468'},\n 'spec': {'active_deadline_seconds': None,\n 'affinity': None,\n 'automount_service_account_token': None,\n 'containers': [{'args': ['--enable-lora',\n '--max-lora-rank=64',\n '--max-loras=2',\n '--max-cpu-loras=4',\n '--lora-modules',\n '\'{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\''],\n 'command': ['/bin/bash',\n '-c',\n 'if [ -f '\n '/etc/profile.d/ibm-aiu-setup.sh '\n ']; then\n'\n ' source '\n '/etc/profile.d/ibm-aiu-setup.sh\n'\n 'fi\n'\n '\n'\n 'if [ "$KSERVE_INFER_ROCE" = '\n '"true" ]; then\n'\n ' echo "Trying to infer RoCE '\n 'configs ... "\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gids/* '\n '2>/dev/null\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gid_attrs/types/* '\n '2>/dev/null\n'\n [e2e-llm-inference-service] '\n'\n ' cat '\n '/proc/driver/nvidia/params\n'\n '\n'\n ' '\n 'KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE '\n 'v2"}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'Discovering active HCAs '\n '..."\n'\n ' active_hcas=()\n'\n ' # Loop through all mlx5 '\n 'devices found in sysfs\n'\n ' for hca_dir in '\n '/sys/class/infiniband/mlx5_*; '\n 'do\n'\n " # Ensure it's a "\n 'directory before proceeding\n'\n ' if [ -d "$hca_dir" ]; '\n 'then\n'\n ' '\n 'hca_name=$(basename '\n '"$hca_dir")\n'\n ' '\n 'port_state_file="$hca_dir/ports/1/state" '\n '# Assume port 1\n'\n ' '\n 'type_file="$hca_dir/ports/1/gid_attrs/types/*"\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'Check if the port state file '\n '${port_state_file} exists '\n 'and contains \'ACTIVE\'"\n'\n ' if [ -f '\n '"$port_state_file" ] && grep '\n '-q "ACTIVE" '\n '"$port_state_file" && grep '\n '-q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '${type_file} 2>/dev/null; '\n 'then\n'\n ' echo "[Infer '\n 'RoCE] Found active HCA: '\n '$hca_name"\n'\n ' '\n 'active_hcas+=("$hca_name")\n'\n ' else\n'\n ' echo "[Infer '\n 'RoCE] Skipping inactive or '\n 'down HCA: $hca_name"\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Check if we found any '\n 'active HCA [e2e-llm-inference-service] s\n'\n ' if [ ${#active_hcas[@]} '\n '-gt 0 ]; then\n'\n ' # Join the array '\n 'elements with a comma\n'\n ' hca_port_pairs=()\n'\n ' for hca in '\n '"${active_hcas[@]}"; do\n'\n ' '\n 'hca_port_pairs+=("${hca}:1")\n'\n ' done\n'\n '\n'\n ' '\n 'active_hca_list=$(IFS=,; '\n 'echo "${active_hcas[*]}")\n'\n ' '\n 'hca_port_pairs_list=$(IFS=,; '\n 'echo '\n '"${hca_port_pairs[*]}")\n'\n ' echo "[Infer RoCE] '\n 'Setting active HCAs: '\n '${active_hca_list}"\n'\n ' export '\n 'NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n'\n ' export '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n'\n ' export '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'NCCL_IB_HCA=${NCCL_IB_HCA}"\n'\n ' echo "[Infer RoCE] '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n'\n ' echo "[Infer RoCE] '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n'\n ' else\n'\n ' echo "[Infer RoCE] '\n 'WARNING: No active RoCE HCAs '\n 'found. NCCL_IB_HCA will not '\n 'be set."\n'\n ' fi\n'\n '\n'\n ' if [ ${#active_hcas[@]} '\n '-gt 0 ]; then\n'\n ' echo "[Infer RoCE] '\n 'Finding GID_INDEX for each '\n 'active HCA (SR-IOV '\n 'compatible)..."\n'\n '\n'\n ' # For SR-IOV '\n 'environments, find the most '\n 'common IPv4 RoCE v2 GID '\n 'index across all HCAs\n'\n ' declare -A '\n 'gid_index_count\n'\n ' declare -A '\n [e2e-llm-inference-service] 'hca_gid_index\n'\n '\n'\n ' for hca_name in '\n '"${active_hcas[@]}"; do\n'\n ' echo "[Infer RoCE] '\n 'Processing HCA: '\n '${hca_name}"\n'\n '\n'\n ' # Find all RoCE v2 '\n 'IPv4 GIDs for this HCA and '\n 'count by index\n'\n ' for tpath in '\n '/sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; '\n 'do\n'\n ' if grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '"$tpath" 2>/dev/null; then\n'\n ' '\n 'idx=$(basename "$tpath")\n'\n ' '\n 'gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n'\n ' # Check '\n 'for IPv4 GID (contains '\n 'ffff:)\n'\n ' if [ -f '\n '"$gid_file" ] && grep -q '\n '"ffff:" "$gid_file"; then\n'\n ' '\n 'gid_value=$(cat "$gid_file" '\n '2>/dev/null || echo "")\n'\n ' echo '\n '"[Infer RoCE] Found IPv4 '\n 'RoCE v2 GID for ${hca_name}: '\n 'index=${idx}, '\n 'gid=${gid_value}"\n'\n ' '\n 'hca_gid_index["${hca_name}"]="${idx}"\n'\n ' '\n 'gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} '\n '+ 1))\n'\n ' break '\n '# Use first found IPv4 GID '\n 'per HCA\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n ' done\n'\n '\n'\n ' # Find the most common '\n 'GID index (most likely to be '\n 'consistent across nodes)\n'\n ' best_gid_index=""\n'\n ' max_count=0\n'\n ' [e2e-llm-inference-service] for idx in '\n '"${!gid_index_count[@]}"; '\n 'do\n'\n ' '\n 'count=${gid_index_count["${idx}"]}\n'\n ' echo "[Infer RoCE] '\n 'GID_INDEX ${idx} found on '\n '${count} HCAs"\n'\n ' if [ $count -gt '\n '$max_count ]; then\n'\n ' '\n 'max_count=$count\n'\n ' '\n 'best_gid_index="$idx"\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Use deterministic '\n 'fallback if tied - prefer '\n 'index 3 (SR-IOV standard)\n'\n ' if [ '\n '${#gid_index_count[@]} -gt 1 '\n ']; then\n'\n ' echo "[Infer RoCE] '\n 'Multiple GID indices found, '\n 'selecting most common: '\n '${best_gid_index}"\n'\n " # If there's a "\n "tie, prefer index 3 as it's "\n 'most common in SR-IOV '\n 'setups\n'\n ' if [ -n '\n '"${gid_index_count[\'3\']}" '\n '] && [ '\n '"${gid_index_count[\'3\']}" '\n '-eq "$max_count" ]; then\n'\n ' '\n 'best_gid_index="3"\n'\n ' echo "[Infer '\n 'RoCE] Using deterministic '\n 'fallback: GID_INDEX=3 '\n '(SR-IOV standard)"\n'\n ' fi\n'\n ' fi\n'\n '\n'\n ' # Check if GID_INDEX '\n 'is already set via '\n 'environment variables\n'\n ' if [ -n '\n '"${NCCL_IB_GID_INDEX}" ]; '\n 'then\n'\n ' echo "[Infer RoCE] '\n 'Using pre-configured '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} '\n 'from environment"\n'\n ' export '\n 'NVSHMEM_IB_G [e2e-llm-inference-service] ID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' echo "[Infer RoCE] '\n 'Using pre-configured '\n 'GID_INDEX=${NCCL_IB_GID_INDEX} '\n 'for NCCL, NVSHMEM, and UCX"\n'\n ' elif [ -n '\n '"$best_gid_index" ]; then\n'\n ' echo "[Infer RoCE] '\n 'Selected GID_INDEX: '\n '${best_gid_index} (found on '\n '${max_count} HCAs)"\n'\n '\n'\n ' export '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'Exported '\n 'GID_INDEX=${best_gid_index} '\n 'for NCCL, NVSHMEM, and UCX"\n'\n ' else\n'\n ' echo "[Infer RoCE] '\n 'ERROR: No valid IPv4 '\n '${KSERVE_INFER_IB_GID_INDEX_GREP} '\n 'GID_INDEX found on any '\n 'HCA."\n'\n ' fi\n'\n ' else\n'\n ' echo "[Infer RoCE] No '\n 'active HCAs found, skipping '\n 'GID_INDEX inference."\n'\n ' fi\n'\n 'fi\n'\n '\n'\n '# '\n '--disable-access-log-for-endpoints '\n 'landed in vLLM 0.16.0 '\n '(vllm-project/vllm#30011).\n'\n '# Older versions still need '\n 'the blanket '\n '--disable-uvicorn-access-log.\n'\n 'ACCESS_LOG_ARGS="--disable-uvicorn-access-log"\n'\n 'VLLM_VERSION=$(vllm '\n '--version 2>/dev/null | tail '\n "-1 | awk '{print $NF}')\n"\n 'echo "[access-log-detect] '\n 'vllm '\n 'version=\'${VLLM_VERSION}\'"\n'\n 'if [[ [e2e-llm-inference-service] "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ '\n '"$(printf \'%s\\n%s\\n\' '\n '"0.16.0" "${VLLM_VERSION}" | '\n 'sort -V | head -1)" = '\n '"0.16.0" ]; then\n'\n ' '\n 'ACCESS_LOG_ARGS="--disable-access-log-for-endpoints '\n '/health,/metrics,/ping"\n'\n 'fi\n'\n 'echo "[access-log-detect] '\n 'selected '\n 'ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n'\n '\n'\n '# --shutdown-timeout landed '\n 'in vLLM 0.18.0 '\n '(vllm-project/vllm#36666).\n'\n 'SHUTDOWN_TIMEOUT_ARGS=""\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ '\n '"$(printf \'%s\\n%s\\n\' '\n '"0.18.0" "${VLLM_VERSION}" | '\n 'sort -V | head -1)" = '\n '"0.18.0" ]; then\n'\n ' '\n 'SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout '\n '40"\n'\n 'fi\n'\n '\n'\n 'eval "exec vllm serve '\n '/mnt/models \\\n'\n ' --served-model-name '\n '"facebook/opt-125m" '\n '"publishers/kserve-ci-e2e-test/models/facebook/opt-125m" '\n '\\\n'\n ' --port 8000 \\\n'\n ' ${ACCESS_LOG_ARGS} \\\n'\n ' ${SHUTDOWN_TIMEOUT_ARGS} '\n '\\\n'\n ' --enable-ssl-refresh \\\n'\n ' --ssl-certfile '\n '/var/run/kserve/tls/tls.crt '\n '\\\n'\n ' --ssl-keyfile '\n '/var/run/kserve/tls/tls.key '\n '\\\n'\n ' ${VLLM_ADDITIONAL_ARGS} '\n '\\\n'\n ' $@"',\n '--'],\n 'env': [{'name': 'HOME',\n 'value': '/home',\n 'value_from': None},\n {'name': 'VLLM_LOGGING_LEVEL',\n 'value': 'DEBUG',\n 'value_from': None},\n {'name': 'VLLM_CPU_KVCACHE_SPACE',\n [e2e-llm-inference-service] 'value': '1',\n 'value_from': None},\n {'name': 'VLLM_ENABLE_V1_MULTIPROCESSING',\n 'value': '0',\n 'value_from': None},\n {'name': 'USER',\n 'value': 'nonroot',\n 'value_from': None},\n {'name': 'TORCHINDUCTOR_CACHE_DIR',\n 'value': '/tmp/torchinductor-cache',\n 'value_from': None},\n {'name': 'HF_HUB_CACHE',\n 'value': '/models',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': {'post_start': None,\n 'pre_stop': {'_exec': {'command': ['/bin/sleep',\n '15']},\n 'http_get': None,\n 'sleep': None,\n 'tcp_socket': None}},\n 'liveness_probe': {'_exec': None,\n 'failure_threshold': 10,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'name': 'main',\n 'ports': [{'container_port': 8000,\n 'host_ip': None,\n 'host_port': None,\n 'name': None,\n 'protocol': 'TCP'}],\n 'readiness_probe': {'_exec': None,\n 'failure_threshold': 2,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_secon [e2e-llm-inference-service] ds': 1,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'resize_policy': None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2',\n 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n 'read_only_root_filesystem': True,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'windows_options': None},\n 'startup_probe': {'_exec': None,\n 'failure_threshold': 60,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/home',\n 'mount_propagation': None,\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None,\n [e2e-llm-inference-service] 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/tmp',\n 'mount_propagation': None,\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/dev/shm',\n 'mount_propagation': None,\n 'name': 'dshm',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/models',\n 'mount_propagation': None,\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/kserve/tls',\n 'mount_propagation': None,\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'dns_config': None,\n 'dns_policy': 'ClusterFirst',\n 'enable_service_links': True,\n 'ephemeral_containers': None,\n 'host_aliases': None,\n 'host_ipc': None,\n 'host_network': None,\n 'host_pid': None,\n 'host_users': None,\n 'hostname': None,\n [e2e-llm-inference-service] 'image_pull_secrets': [{'name': 'default-dockercfg-6qcsh'}],\n 'init_containers': [{'args': ['hf://facebook/opt-125m',\n '/mnt/models',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-1',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-2'],\n 'command': None,\n 'env': [{'name': 'AWS_ACCESS_KEY_ID',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': None,\n 'secret_key_ref': {'key': 'AWS_ACCESS_KEY_ID',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'AWS_SECRET_ACCESS_KEY',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': None,\n 'secret_key_ref': {'key': 'AWS_SECRET_ACCESS_KEY',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'S3_USE_HTTPS',\n 'value': '0',\n 'value_from': None},\n {'name': 'S3_ENDPOINT',\n 'value': 's3-service.kserve:8333',\n 'value_from': None},\n {'name': 'AWS_ENDPOINT_URL',\n 'value': 'http://s3-service.kserve:8333',\n 'value_from': None},\n {'name': 'S3_VERIFY_SSL',\n 'value': '0',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE',\n 'value': '/etc/ssl/custom-certs/cabundle.crt',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE_CONFIGMAP',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'HF_HUB_ENABLE_HF_TRANSFER',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_HIGH_PERFORMANCE',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_NUM_CONCURRENT_RANGE_GETS', [e2e-llm-inference-service] \n 'value': '8',\n 'value_from': None},\n {'name': 'CA_BUNDLE_CONFIGMAP_NAME',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'CA_BUNDLE_VOLUME_MOUNT_POINT',\n 'value': '/etc/ssl/custom-certs',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': None,\n 'liveness_probe': None,\n 'name': 'storage-initializer',\n 'ports': None,\n 'readiness_probe': None,\n 'resize_policy': None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n 'read_only_root_filesystem': None,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': None,\n 'windows_options': None},\n 'startup_probe': None,\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n [e2e-llm-inference-service] 'sub_path_expr': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'mount_propagation': None,\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'node_name': 'ip-10-0-131-57.ec2.internal',\n 'node_selector': None,\n 'os': None,\n 'overhead': None,\n 'preemption_policy': 'PreemptLowerPriority',\n 'priority': 0,\n 'priority_class_name': None,\n 'readiness_gates': None,\n 'resource_claims': None,\n 'resources': None,\n 'restart_policy': 'Always',\n 'runtime_class_name': None,\n 'scheduler_name': 'default-scheduler',\n 'scheduling_gates': None,\n 'security_context': {'app_armor_profile': None,\n 'fs_group': 1000690000,\n 'fs_group_change_policy': None,\n 'run_as_group': None,\n 'run_as_non_root': None,\n 'run_as_user': None,\n 'se_linux_change_policy': None,\n 'se_linux_options': {'level': 's0:c26,c20',\n 'role': None,\n 'type': None,\n 'user': None},\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'supplemental_groups': None,\n 'supplemental_groups_policy': None,\n 'sysctls': None,\n 'windows_options': None},\n 'service_account': 'default',\n 'service_account_name': 'default',\n 'set_hostname_as_fqdn': None,\n 'share_process_namespace': None,\n 'subdomain': None,\n 'termination_grace_period_seconds': 60,\n 'tolerations': [{'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/not-ready',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/unreac [e2e-llm-inference-service] hable',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoSchedule',\n 'key': 'node.kubernetes.io/memory-pressure',\n 'operator': 'Exists',\n 'toleration_seconds': None,\n 'value': None}],\n 'topology_spread_constraints': None,\n 'volumes': [{'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None,\n 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'home',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': 'Memory',\n 'size_limit': '1Gi'},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'dshm',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n [e2e-llm-inference-service] 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None,\n 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'model-cache',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None,\n 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tmp-dir',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n [e2e-llm-inference-service] 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tls-certs',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': {'default_mode': 420,\n 'items': None,\n 'optional': None,\n 'secret_name': 'lora-multiple-adapters-test-kserve-self-signed-certs'},\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None,\n 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kserve-provision-location',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': {'default_mode': 420,\n 'items': None,\n 'name': 'odh-kserve-custom-ca-bundle',\n 'optional': None},\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n [e2e-llm-inference-service] 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'cabundle-cert',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kube-api-access-n5dws',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': {'default_mode': 420,\n 'sources': [{'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': {'audience': None,\n 'expiration_seconds': 3607,\n 'path': 'token'}},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'ca.crt',\n 'mode': None,\n 'path': 'ca.crt'}],\n 'name': 'kube-root-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None},\n [e2e-llm-inference-service] {'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': {'items': [{'field_ref': {'api_version': 'v1',\n 'field_path': 'metadata.namespace'},\n 'mode': None,\n 'path': 'namespace',\n 'resource_field_ref': None}]},\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'service-ca.crt',\n 'mode': None,\n 'path': 'service-ca.crt'}],\n 'name': 'openshift-service-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None}]},\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None}]},\n 'status': {'conditions': [{'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 18, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodReadyToStartContainers'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'Initialized'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'Ready'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'ContainersReady'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'message': None,\n [e2e-llm-inference-service] 'reason': None,\n 'status': 'True',\n 'type': 'PodScheduled'}],\n 'container_statuses': [{'allocated_resources': {'cpu': '200m',\n 'memory': '2Gi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402',\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_id': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'main',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2',\n 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_count': 0,\n 'started': True,\n 'state': {'running': {'started_at': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal())},\n 'terminated': None,\n 'waiting': None},\n 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/home',\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/tmp',\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/dev/shm',\n 'name': 'dshm',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/models',\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/var/run/kserve/tls',\n 'name': 'tls-certs',\n [e2e-llm-inference-service] 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'ephemeral_container_statuses': None,\n 'host_i_ps': [{'ip': '10.0.131.57'}],\n 'host_ip': '10.0.131.57',\n 'init_container_statuses': [{'allocated_resources': {'cpu': '100m',\n 'memory': '100Mi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_id': 'quay.io/opendatahub/kserve-storage-initializer@sha256:60e2a5c8616d82c8933b614ae003dd3412a420a29a0be570b6a6439a3d39ab99',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'storage-initializer',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_count': 0,\n 'started': False,\n 'state': {'running': None,\n 'terminated': {'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'exit_code': 0,\n 'finished_at': datetime.datetime(2026, 7, 6, 21, 35, 22, tzinfo=tzlocal()),\n 'message': None,\n 'reason': 'Completed',\n 'signal': None,\n 'started_at': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n 'waiting': None},\n 'user': {'linux': {'gid': 0,\n [e2e-llm-inference-service] 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'message': None,\n 'nominated_node_name': None,\n 'phase': 'Running',\n 'pod_i_ps': [{'ip': '10.134.0.76'}],\n 'pod_ip': '10.134.0.76',\n 'qos_class': 'Burstable',\n 'reason': None,\n 'resize': None,\n 'resource_claim_statuses': None,\n 'start_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())}}],\n 'kind': 'PodList',\n 'metadata': {'_continue': None,\n 'remaining_item_count': None,\n 'resource_version': '79752',\n 'self_link': None}}.items [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: 1 pod(s) for lora-multiple-adapters-test still terminating [e2e-llm-inference-service] assert not [{'api_version': None,\n 'kind': None,\n 'metadata': {'annotations': {'k8s.ovn.org/pod-networks': '{"default":{"ip_addresses":["10.134.0.76/23"],"mac_address":"0a:58:0a:86:00:4c","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.76/23","gateway_ip":"10.134.0.1","role":"primary"}}',\n 'k8s.v1.cni.cncf.io/network-status': '[{\n'\n ' '\n '"name": '\n '"ovn-kubernetes",\n'\n ' '\n '"interface": '\n '"eth0",\n'\n ' "ips": '\n '[\n'\n ' '\n '"10.134.0.76"\n'\n ' ],\n'\n ' "mac": '\n '"0a:58:0a:86:00:4c",\n'\n ' '\n '"default": '\n 'true,\n'\n ' "dns": '\n '{}\n'\n '}]',\n 'openshift.io/scc': 'restricted-v2',\n 'seccomp.security.alpha.kubernetes.io/pod': 'runtime/default',\n 'security.openshift.io/validated-scc-subject-type': 'user'},\n 'creation_timestamp': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'deletion_grace_period_seconds': 60,\n 'deletion_timestamp': datetime.datetime(2026, 7, 6, 21, 52, 53, tzinfo=tzlocal()),\n 'finalizers': None,\n 'generate_name': 'lora-multiple-adapters-test-kserve-77485bff95-',\n 'generation': 2,\n 'labels': {'app.kubernetes.io/component': 'llminferenceservice-workload',\n 'app.kubernetes.io/name': 'lora-multiple-adapters-test',\n 'app.kubernetes.io/part-of': 'llminferenceservice',\n 'kserve.io/component': 'workload',\n 'llm-d.ai/role': 'both',\n 'pod-template-hash': '77485bff95'},\n 'managed_fields': [{'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.ovn.org/pod-networks': {}}}},\n 'manager': 'ip-10-0-131-57',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:generateName': {},\n 'f:labels': {'.': {},\n 'f:app.kubernetes.io/component' [e2e-llm-inference-service] : {},\n 'f:app.kubernetes.io/name': {},\n 'f:app.kubernetes.io/part-of': {},\n 'f:kserve.io/component': {},\n 'f:llm-d.ai/role': {},\n 'f:pod-template-hash': {}},\n 'f:ownerReferences': {'.': {},\n 'k:{"uid":"75c381c4-1742-4fb9-b05f-92ea6db35c9c"}': {}}},\n 'f:spec': {'f:containers': {'k:{"name":"main"}': {'.': {},\n 'f:args': {},\n 'f:command': {},\n 'f:env': {'.': {},\n 'k:{"name":"HF_HUB_CACHE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HOME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"TORCHINDUCTOR_CACHE_DIR"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"USER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_CPU_KVCACHE_SPACE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n [e2e-llm-inference-service] 'k:{"name":"VLLM_LOGGING_LEVEL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:lifecycle': {'.': {},\n 'f:preStop': {'.': {},\n 'f:exec': {'.': {},\n 'f:command': {}}}},\n 'f:livenessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:name': {},\n 'f:ports': {'.': {},\n 'k:{"containerPort":8000,"protocol":"TCP"}': {'.': {},\n 'f:containerPort': {},\n 'f:protocol': {}}},\n 'f:readinessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n [e2e-llm-inference-service] 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:securityContext': {'.': {},\n 'f:allowPrivilegeEscalation': {},\n 'f:capabilities': {'.': {},\n 'f:drop': {}},\n 'f:readOnlyRootFilesystem': {},\n 'f:runAsNonRoot': {},\n 'f:seccompProfile': {'.': {},\n 'f:type': {}}},\n 'f:startupProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:terminationMessagePath': {},\n [e2e-llm-inference-service] 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/dev/shm"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/home"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/models"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/tmp"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/var/run/kserve/tls"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}}}}},\n 'f:dnsPolicy': {},\n 'f:enableServiceLinks': {},\n 'f:initContainers': {'.': {},\n 'k:{"name":"storage-initializer"}': {'.': {},\n 'f:args': {},\n [e2e-llm-inference-service] 'f:env': {'.': {},\n 'k:{"name":"AWS_ACCESS_KEY_ID"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"AWS_CA_BUNDLE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_ENDPOINT_URL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_SECRET_ACCESS_KEY"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}': {'.': {},\n 'f:name': {},\n [e2e-llm-inference-service] 'f:value': {}},\n 'k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_HIGH_PERFORMANCE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_ENDPOINT"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_USE_HTTPS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_VERIFY_SSL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:name': {},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n [e2e-llm-inference-service] 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:terminationMessagePath': {},\n 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/etc/ssl/custom-certs"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}}}}},\n 'f:restartPolicy': {},\n 'f:schedulerName': {},\n 'f:securityContext': {},\n 'f:terminationGracePeriodSeconds': {},\n 'f:volumes': {'.': {},\n 'k:{"name":"cabundle-cert"}': {'.': {},\n 'f:configMap': {'.': {},\n 'f:defaultMode': {},\n 'f:name': {}},\n 'f:name': {}},\n 'k:{"name":"dshm"}': {'.': {},\n 'f:emptyDir': {'.': {},\n 'f:medium': {},\n [e2e-llm-inference-service] 'f:sizeLimit': {}},\n 'f:name': {}},\n 'k:{"name":"home"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"kserve-provision-location"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"model-cache"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"tls-certs"}': {'.': {},\n 'f:name': {},\n 'f:secret': {'.': {},\n 'f:defaultMode': {},\n 'f:secretName': {}}},\n 'k:{"name":"tmp-dir"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}}}}},\n 'manager': 'kube-controller-manager',\n 'operation': 'Update',\n 'subresource': None,\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.v1.cni.cncf.io/network-status': {}}}},\n 'manager': 'multus-daemon',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:status': {'f:conditions': {'k:{"type":"ContainersReady"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"Initialized"}': {'.': {},\n [e2e-llm-inference-service] 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodReadyToStartContainers"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodScheduled"}': {'f:observedGeneration': {}},\n 'k:{"type":"Ready"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}}},\n 'f:containerStatuses': {},\n 'f:hostIP': {},\n 'f:hostIPs': {},\n 'f:initContainerStatuses': {},\n 'f:observedGeneration': {},\n 'f:phase': {},\n 'f:podIP': {},\n 'f:podIPs': {'.': {},\n 'k:{"ip":"10.134.0.76"}': {'.': {},\n 'f:ip': {}}},\n 'f:startTime': {}}},\n 'manager': 'kubelet',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 51, 53, tzinfo=tzlocal())}],\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95-2tx5t',\n 'namespace': 'kserve-ci-e2e-test',\n 'owner_references': [{'api_version': 'apps/v1',\n 'block_owner_deletion': True,\n 'controller': True,\n 'kind': 'ReplicaSet',\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95',\n 'uid': '75c381c4-1742-4fb9-b05f-92ea6db3 [e2e-llm-inference-service] 5c9c'}],\n 'resource_version': '79769',\n 'self_link': None,\n 'uid': 'd4948f86-f0d1-4911-908f-7cb88db94468'},\n 'spec': {'active_deadline_seconds': None,\n 'affinity': None,\n 'automount_service_account_token': None,\n 'containers': [{'args': ['--enable-lora',\n '--max-lora-rank=64',\n '--max-loras=2',\n '--max-cpu-loras=4',\n '--lora-modules',\n '\'{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\''],\n 'command': ['/bin/bash',\n '-c',\n 'if [ -f /etc/profile.d/ibm-aiu-setup.sh '\n ']; then\n'\n ' source '\n '/etc/profile.d/ibm-aiu-setup.sh\n'\n 'fi\n'\n '\n'\n 'if [ "$KSERVE_INFER_ROCE" = "true" ]; '\n 'then\n'\n ' echo "Trying to infer RoCE configs '\n '... "\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gids/* '\n '2>/dev/null\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gid_attrs/types/* '\n '2>/dev/null\n'\n '\n'\n ' cat /proc/driver/nvidia/params\n'\n '\n'\n ' '\n 'KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE '\n 'v2"}\n'\n '\n'\n ' echo "[Infer RoCE] Discovering active '\n 'HCAs ..."\n'\n ' active_hcas=()\n'\n ' # Loop through all mlx5 devices found '\n 'in sysfs\n'\n ' for hca_dir in '\n '/sys/class/infiniband/mlx5_*; do\n'\n " # Ensure it's a directory before "\n 'proceeding\n'\n ' if [ -d "$hca_dir" ]; then\n'\n ' hca_name=$(basename '\n '"$hca_dir")\n'\n ' '\n 'port_state_file="$hca_dir/ports/1/state" '\n '# Assume port 1\n'\n ' '\n 'type_file="$hca_dir/ports/1/gid_attrs/types/*"\n'\n '\n'\n ' echo "[Infer RoCE] Check if '\n 'the port state file ${port_state_file} '\n 'exists and contains \'ACTIVE\'"\n'\n ' if [ -f "$port_state_file" ] '\n '&& grep -q "AC [e2e-llm-inference-service] TIVE" "$port_state_file" '\n '&& grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '${type_file} 2>/dev/null; then\n'\n ' echo "[Infer RoCE] Found '\n 'active HCA: $hca_name"\n'\n ' '\n 'active_hcas+=("$hca_name")\n'\n ' else\n'\n ' echo "[Infer RoCE] '\n 'Skipping inactive or down HCA: '\n '$hca_name"\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Check if we found any active HCAs\n'\n ' if [ ${#active_hcas[@]} -gt 0 ]; '\n 'then\n'\n ' # Join the array elements with a '\n 'comma\n'\n ' hca_port_pairs=()\n'\n ' for hca in "${active_hcas[@]}"; '\n 'do\n'\n ' hca_port_pairs+=("${hca}:1")\n'\n ' done\n'\n '\n'\n ' active_hca_list=$(IFS=,; echo '\n '"${active_hcas[*]}")\n'\n ' hca_port_pairs_list=$(IFS=,; echo '\n '"${hca_port_pairs[*]}")\n'\n ' echo "[Infer RoCE] Setting active '\n 'HCAs: ${active_hca_list}"\n'\n ' export '\n 'NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n'\n ' export '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n'\n ' export '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'NCCL_IB_HCA=${NCCL_IB_HCA}"\n'\n ' echo "[Infer RoCE] '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n'\n ' echo "[Infer RoCE] '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n'\n ' else\n'\n ' echo "[Infer RoCE] WARNING: No '\n 'active RoCE HCAs found. NCCL_IB_HCA '\n 'will not be set."\n'\n ' fi\n'\n '\n'\n ' if [ ${#active_hcas[@]} -gt 0 ]; '\n 'then\n'\n ' echo "[Infer RoCE] Finding '\n 'GID_INDEX for each active HCA (SR-IOV '\n 'compatible)..."\n'\n '\n'\n ' # For SR-IOV environments, find '\n 'the most common IPv4 RoCE v2 GID index '\n 'across all HCAs\n'\n ' declare -A gid_index_count\n'\n [e2e-llm-inference-service] ' declare -A hca_gid_index\n'\n '\n'\n ' for hca_name in '\n '"${active_hcas[@]}"; do\n'\n ' echo "[Infer RoCE] Processing '\n 'HCA: ${hca_name}"\n'\n '\n'\n ' # Find all RoCE v2 IPv4 GIDs '\n 'for this HCA and count by index\n'\n ' for tpath in '\n '/sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; '\n 'do\n'\n ' if grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '"$tpath" 2>/dev/null; then\n'\n ' idx=$(basename '\n '"$tpath")\n'\n ' '\n 'gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n'\n ' # Check for IPv4 GID '\n '(contains ffff:)\n'\n ' if [ -f "$gid_file" ] '\n '&& grep -q "ffff:" "$gid_file"; then\n'\n ' gid_value=$(cat '\n '"$gid_file" 2>/dev/null || echo "")\n'\n ' echo "[Infer '\n 'RoCE] Found IPv4 RoCE v2 GID for '\n '${hca_name}: index=${idx}, '\n 'gid=${gid_value}"\n'\n ' '\n 'hca_gid_index["${hca_name}"]="${idx}"\n'\n ' '\n 'gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} '\n '+ 1))\n'\n ' break # Use '\n 'first found IPv4 GID per HCA\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n ' done\n'\n '\n'\n ' # Find the most common GID index '\n '(most likely to be consistent across '\n 'nodes)\n'\n ' best_gid_index=""\n'\n ' max_count=0\n'\n ' for idx in '\n '"${!gid_index_count[@]}"; do\n'\n ' '\n 'count=${gid_index_count["${idx}"]}\n'\n ' echo "[Infer RoCE] GID_INDEX '\n '${idx} found on ${count} HCAs"\n'\n ' if [ $count -gt $max_count ]; '\n 'then\n'\n ' max_count=$count\n'\n ' best_gid_index="$idx"\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Use determini [e2e-llm-inference-service] stic fallback if '\n 'tied - prefer index 3 (SR-IOV '\n 'standard)\n'\n ' if [ ${#gid_index_count[@]} -gt 1 '\n ']; then\n'\n ' echo "[Infer RoCE] Multiple '\n 'GID indices found, selecting most '\n 'common: ${best_gid_index}"\n'\n " # If there's a tie, prefer "\n "index 3 as it's most common in SR-IOV "\n 'setups\n'\n ' if [ -n '\n '"${gid_index_count[\'3\']}" ] && [ '\n '"${gid_index_count[\'3\']}" -eq '\n '"$max_count" ]; then\n'\n ' best_gid_index="3"\n'\n ' echo "[Infer RoCE] Using '\n 'deterministic fallback: GID_INDEX=3 '\n '(SR-IOV standard)"\n'\n ' fi\n'\n ' fi\n'\n '\n'\n ' # Check if GID_INDEX is already '\n 'set via environment variables\n'\n ' if [ -n "${NCCL_IB_GID_INDEX}" ]; '\n 'then\n'\n ' echo "[Infer RoCE] Using '\n 'pre-configured '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} '\n 'from environment"\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' echo "[Infer RoCE] Using '\n 'pre-configured '\n 'GID_INDEX=${NCCL_IB_GID_INDEX} for '\n 'NCCL, NVSHMEM, and UCX"\n'\n ' elif [ -n "$best_gid_index" ]; '\n 'then\n'\n ' echo "[Infer RoCE] Selected '\n 'GID_INDEX: ${best_gid_index} (found on '\n '${max_count} HCAs)"\n'\n '\n'\n ' export '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n'\n '\n'\n ' echo "[Infer RoCE] Exported '\n 'GID_INDEX=${best_gid_index} for NCCL, '\n 'NVSHMEM, and UCX"\n'\n ' else\n'\n ' echo "[Infer RoCE] ERROR: No '\n 'valid IPv4 '\n '${KSERVE_INFER_IB_GID_INDEX_GREP} '\n 'GID_INDEX found on any HCA."\n'\n [e2e-llm-inference-service] ' fi\n'\n ' else\n'\n ' echo "[Infer RoCE] No active HCAs '\n 'found, skipping GID_INDEX inference."\n'\n ' fi\n'\n 'fi\n'\n '\n'\n '# --disable-access-log-for-endpoints '\n 'landed in vLLM 0.16.0 '\n '(vllm-project/vllm#30011).\n'\n '# Older versions still need the blanket '\n '--disable-uvicorn-access-log.\n'\n 'ACCESS_LOG_ARGS="--disable-uvicorn-access-log"\n'\n 'VLLM_VERSION=$(vllm --version '\n "2>/dev/null | tail -1 | awk '{print "\n "$NF}')\n"\n 'echo "[access-log-detect] vllm '\n 'version=\'${VLLM_VERSION}\'"\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ "$(printf '\n '\'%s\\n%s\\n\' "0.16.0" '\n '"${VLLM_VERSION}" | sort -V | head -1)" '\n '= "0.16.0" ]; then\n'\n ' '\n 'ACCESS_LOG_ARGS="--disable-access-log-for-endpoints '\n '/health,/metrics,/ping"\n'\n 'fi\n'\n 'echo "[access-log-detect] selected '\n 'ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n'\n '\n'\n '# --shutdown-timeout landed in vLLM '\n '0.18.0 (vllm-project/vllm#36666).\n'\n 'SHUTDOWN_TIMEOUT_ARGS=""\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ "$(printf '\n '\'%s\\n%s\\n\' "0.18.0" '\n '"${VLLM_VERSION}" | sort -V | head -1)" '\n '= "0.18.0" ]; then\n'\n ' '\n 'SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout '\n '40"\n'\n 'fi\n'\n '\n'\n 'eval "exec vllm serve /mnt/models \\\n'\n ' --served-model-name '\n '"facebook/opt-125m" '\n '"publishers/kserve-ci-e2e-test/models/facebook/opt-125m" '\n '\\\n'\n ' --port 8000 \\\n'\n ' ${ACCESS_LOG_ARGS} \\\n'\n ' ${SHUTDOWN_TIMEOUT_ARGS} \\\n'\n ' --enable-ssl-refresh \\\n'\n ' --ssl-certfile '\n '/var/run/kserve/tls/tls.crt \\\n'\n ' --ssl-keyfile '\n '/var/run/kserve/tls/tls.key \\\n'\n ' ${VLLM_ADDITIONAL_ARGS} \\\n'\n ' $@"',\n '--'],\n 'env': [{'name': 'HOME',\n 'value': '/home',\n 'value_from': None},\n {'name': 'VLL [e2e-llm-inference-service] M_LOGGING_LEVEL',\n 'value': 'DEBUG',\n 'value_from': None},\n {'name': 'VLLM_CPU_KVCACHE_SPACE',\n 'value': '1',\n 'value_from': None},\n {'name': 'VLLM_ENABLE_V1_MULTIPROCESSING',\n 'value': '0',\n 'value_from': None},\n {'name': 'USER',\n 'value': 'nonroot',\n 'value_from': None},\n {'name': 'TORCHINDUCTOR_CACHE_DIR',\n 'value': '/tmp/torchinductor-cache',\n 'value_from': None},\n {'name': 'HF_HUB_CACHE',\n 'value': '/models',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': {'post_start': None,\n 'pre_stop': {'_exec': {'command': ['/bin/sleep',\n '15']},\n 'http_get': None,\n 'sleep': None,\n 'tcp_socket': None}},\n 'liveness_probe': {'_exec': None,\n 'failure_threshold': 10,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'name': 'main',\n 'ports': [{'container_port': 8000,\n 'host_ip': None,\n 'host_port': None,\n 'name': None,\n 'protocol': 'TCP'}],\n 'readiness_probe': {'_exec': None,\n 'failure_threshold': 2,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 1,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'resize_policy': [e2e-llm-inference-service] None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2', 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n 'read_only_root_filesystem': True,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'windows_options': None},\n 'startup_probe': {'_exec': None,\n 'failure_threshold': 60,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/home',\n 'mount_propagation': None,\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/tmp',\n 'mount_propagation': None,\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/dev/shm',\n 'mount_propagation': None,\n 'name': 'dshm',\n 'read_only': None,\n 'recursive_read_only': None,\n [e2e-llm-inference-service] 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/models',\n 'mount_propagation': None,\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/kserve/tls',\n 'mount_propagation': None,\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'dns_config': None,\n 'dns_policy': 'ClusterFirst',\n 'enable_service_links': True,\n 'ephemeral_containers': None,\n 'host_aliases': None,\n 'host_ipc': None,\n 'host_network': None,\n 'host_pid': None,\n 'host_users': None,\n 'hostname': None,\n 'image_pull_secrets': [{'name': 'default-dockercfg-6qcsh'}],\n 'init_containers': [{'args': ['hf://facebook/opt-125m',\n '/mnt/models',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-1',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-2'],\n 'command': None,\n 'env': [{'name': 'AWS_ACCESS_KEY_ID',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': None,\n 'secret_key_ref': {'key': 'AWS_ACCESS_KEY_ID',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'AWS_SECRET_ACCESS_KEY',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': [e2e-llm-inference-service] None,\n 'secret_key_ref': {'key': 'AWS_SECRET_ACCESS_KEY',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'S3_USE_HTTPS',\n 'value': '0',\n 'value_from': None},\n {'name': 'S3_ENDPOINT',\n 'value': 's3-service.kserve:8333',\n 'value_from': None},\n {'name': 'AWS_ENDPOINT_URL',\n 'value': 'http://s3-service.kserve:8333',\n 'value_from': None},\n {'name': 'S3_VERIFY_SSL',\n 'value': '0',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE',\n 'value': '/etc/ssl/custom-certs/cabundle.crt',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE_CONFIGMAP',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'HF_HUB_ENABLE_HF_TRANSFER',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_HIGH_PERFORMANCE',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_NUM_CONCURRENT_RANGE_GETS',\n 'value': '8',\n 'value_from': None},\n {'name': 'CA_BUNDLE_CONFIGMAP_NAME',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'CA_BUNDLE_VOLUME_MOUNT_POINT',\n 'value': '/etc/ssl/custom-certs',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': None,\n 'liveness_probe': None,\n 'name': 'storage-initializer',\n 'ports': None,\n 'readiness_probe': None,\n 'resize_policy': None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n [e2e-llm-inference-service] 'read_only_root_filesystem': None,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': None,\n 'windows_options': None},\n 'startup_probe': None,\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'mount_propagation': None,\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'node_name': 'ip-10-0-131-57.ec2.internal',\n 'node_selector': None,\n 'os': None,\n 'overhead': None,\n 'preemption_policy': 'PreemptLowerPriority',\n 'priority': 0,\n 'priority_class_name': None,\n 'readiness_gates': None,\n 'resource_claims': None,\n 'resources': None,\n 'restart_policy': 'Always',\n 'runtime_class_name': None,\n 'scheduler_name': 'default-scheduler',\n 'scheduling_gates': None,\n 'security_context': {'app_armor_profile': None,\n 'fs_group': 1000690000,\n 'fs_group_change_policy': None,\n 'run_as_group': None,\n 'run_as_non_root': None,\n 'run_as_user': None,\n 'se_linux_change_policy': None,\n 'se_linux_options': {'level': 's0:c26,c20',\n 'role': None,\n 'type': None,\n 'user': None},\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'supplemental_groups': None,\n [e2e-llm-inference-service] 'supplemental_groups_policy': None,\n 'sysctls': None,\n 'windows_options': None},\n 'service_account': 'default',\n 'service_account_name': 'default',\n 'set_hostname_as_fqdn': None,\n 'share_process_namespace': None,\n 'subdomain': None,\n 'termination_grace_period_seconds': 60,\n 'tolerations': [{'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/not-ready',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/unreachable',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoSchedule',\n 'key': 'node.kubernetes.io/memory-pressure',\n 'operator': 'Exists',\n 'toleration_seconds': None,\n 'value': None}],\n 'topology_spread_constraints': None,\n 'volumes': [{'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'home',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': 'Memory', 'size_limit': '1Gi'},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'dshm',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_ [e2e-llm-inference-service] block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'model-cache',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tmp-dir',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tls-certs',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': {'default_mode': 420,\n 'items': None,\n [e2e-llm-inference-service] 'optional': None,\n 'secret_name': 'lora-multiple-adapters-test-kserve-self-signed-certs'},\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kserve-provision-location',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': {'default_mode': 420,\n 'items': None,\n 'name': 'odh-kserve-custom-ca-bundle',\n 'optional': None},\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'cabundle-cert',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kube-api-access-n5dws',\n [e2e-llm-inference-service] 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': {'default_mode': 420,\n 'sources': [{'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': {'audience': None,\n 'expiration_seconds': 3607,\n 'path': 'token'}},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'ca.crt',\n 'mode': None,\n 'path': 'ca.crt'}],\n 'name': 'kube-root-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': {'items': [{'field_ref': {'api_version': 'v1',\n 'field_path': 'metadata.namespace'},\n 'mode': None,\n 'path': 'namespace',\n 'resource_field_ref': None}]},\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'service-ca.crt',\n 'mode': None,\n 'path': 'service-ca.crt'}],\n 'name': 'openshift-service-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None}]},\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None}]},\n 'status': {'conditions': [{'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 18, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodReadyToStartContainers'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal()),\n 'message': None,\n [e2e-llm-inference-service] 'reason': None,\n 'status': 'True',\n 'type': 'Initialized'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'Ready'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'ContainersReady'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodScheduled'}],\n 'container_statuses': [{'allocated_resources': {'cpu': '200m',\n 'memory': '2Gi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402',\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_id': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'main',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2',\n 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_count': 0,\n 'started': True,\n 'state': {'running': {'started_at': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal())},\n 'terminated': None,\n 'waiting': None},\n 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/home',\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/tmp',\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/dev/shm',\n 'name': 'dshm',\n 'read_only': None,\n [e2e-llm-inference-service] 'recursive_read_only': None},\n {'mount_path': '/models',\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/var/run/kserve/tls',\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'ephemeral_container_statuses': None,\n 'host_i_ps': [{'ip': '10.0.131.57'}],\n 'host_ip': '10.0.131.57',\n 'init_container_statuses': [{'allocated_resources': {'cpu': '100m',\n 'memory': '100Mi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_id': 'quay.io/opendatahub/kserve-storage-initializer@sha256:60e2a5c8616d82c8933b614ae003dd3412a420a29a0be570b6a6439a3d39ab99',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'storage-initializer',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_count': 0,\n 'started': False,\n 'state': {'running': None,\n 'terminated': {'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'exit_code': 0,\n 'finished_at': datetime.datetime(2026, 7, 6, 21, 35, 22, tzinfo=tzlocal()),\n 'message': None,\n 'reason': 'Completed',\n 'signal': None,\n 'started_at': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n [e2e-llm-inference-service] 'waiting': None},\n 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'message': None,\n 'nominated_node_name': None,\n 'phase': 'Running',\n 'pod_i_ps': [{'ip': '10.134.0.76'}],\n 'pod_ip': '10.134.0.76',\n 'qos_class': 'Burstable',\n 'reason': None,\n 'resize': None,\n 'resource_claim_statuses': None,\n 'start_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())}}] [e2e-llm-inference-service] + where [{'api_version': None,\n 'kind': None,\n 'metadata': {'annotations': {'k8s.ovn.org/pod-networks': '{"default":{"ip_addresses":["10.134.0.76/23"],"mac_address":"0a:58:0a:86:00:4c","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.76/23","gateway_ip":"10.134.0.1","role":"primary"}}',\n 'k8s.v1.cni.cncf.io/network-status': '[{\n'\n ' '\n '"name": '\n '"ovn-kubernetes",\n'\n ' '\n '"interface": '\n '"eth0",\n'\n ' "ips": '\n '[\n'\n ' '\n '"10.134.0.76"\n'\n ' ],\n'\n ' "mac": '\n '"0a:58:0a:86:00:4c",\n'\n ' '\n '"default": '\n 'true,\n'\n ' "dns": '\n '{}\n'\n '}]',\n 'openshift.io/scc': 'restricted-v2',\n 'seccomp.security.alpha.kubernetes.io/pod': 'runtime/default',\n 'security.openshift.io/validated-scc-subject-type': 'user'},\n 'creation_timestamp': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'deletion_grace_period_seconds': 60,\n 'deletion_timestamp': datetime.datetime(2026, 7, 6, 21, 52, 53, tzinfo=tzlocal()),\n 'finalizers': None,\n 'generate_name': 'lora-multiple-adapters-test-kserve-77485bff95-',\n 'generation': 2,\n 'labels': {'app.kubernetes.io/component': 'llminferenceservice-workload',\n 'app.kubernetes.io/name': 'lora-multiple-adapters-test',\n 'app.kubernetes.io/part-of': 'llminferenceservice',\n 'kserve.io/component': 'workload',\n 'llm-d.ai/role': 'both',\n 'pod-template-hash': '77485bff95'},\n 'managed_fields': [{'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.ovn.org/pod-networks': {}}}},\n 'manager': 'ip-10-0-131-57',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:generateName': {},\n 'f:labels': {'.': {},\n 'f:app.kubernetes.io/component': [e2e-llm-inference-service] {},\n 'f:app.kubernetes.io/name': {},\n 'f:app.kubernetes.io/part-of': {},\n 'f:kserve.io/component': {},\n 'f:llm-d.ai/role': {},\n 'f:pod-template-hash': {}},\n 'f:ownerReferences': {'.': {},\n 'k:{"uid":"75c381c4-1742-4fb9-b05f-92ea6db35c9c"}': {}}},\n 'f:spec': {'f:containers': {'k:{"name":"main"}': {'.': {},\n 'f:args': {},\n 'f:command': {},\n 'f:env': {'.': {},\n 'k:{"name":"HF_HUB_CACHE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HOME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"TORCHINDUCTOR_CACHE_DIR"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"USER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_CPU_KVCACHE_SPACE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n [e2e-llm-inference-service] 'k:{"name":"VLLM_LOGGING_LEVEL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:lifecycle': {'.': {},\n 'f:preStop': {'.': {},\n 'f:exec': {'.': {},\n 'f:command': {}}}},\n 'f:livenessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:name': {},\n 'f:ports': {'.': {},\n 'k:{"containerPort":8000,"protocol":"TCP"}': {'.': {},\n 'f:containerPort': {},\n 'f:protocol': {}}},\n 'f:readinessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n [e2e-llm-inference-service] 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:securityContext': {'.': {},\n 'f:allowPrivilegeEscalation': {},\n 'f:capabilities': {'.': {},\n 'f:drop': {}},\n 'f:readOnlyRootFilesystem': {},\n 'f:runAsNonRoot': {},\n 'f:seccompProfile': {'.': {},\n 'f:type': {}}},\n 'f:startupProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:terminationMessagePath': {},\n [e2e-llm-inference-service] 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/dev/shm"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/home"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/models"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/tmp"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/var/run/kserve/tls"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}}}}},\n 'f:dnsPolicy': {},\n 'f:enableServiceLinks': {},\n 'f:initContainers': {'.': {},\n 'k:{"name":"storage-initializer"}': {'.': {},\n 'f:args': {},\n [e2e-llm-inference-service] 'f:env': {'.': {},\n 'k:{"name":"AWS_ACCESS_KEY_ID"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"AWS_CA_BUNDLE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_ENDPOINT_URL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_SECRET_ACCESS_KEY"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}': {'.': {},\n 'f:name': {},\n [e2e-llm-inference-service] 'f:value': {}},\n 'k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_HIGH_PERFORMANCE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_ENDPOINT"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_USE_HTTPS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_VERIFY_SSL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:name': {},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n [e2e-llm-inference-service] 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:terminationMessagePath': {},\n 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/etc/ssl/custom-certs"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}}}}},\n 'f:restartPolicy': {},\n 'f:schedulerName': {},\n 'f:securityContext': {},\n 'f:terminationGracePeriodSeconds': {},\n 'f:volumes': {'.': {},\n 'k:{"name":"cabundle-cert"}': {'.': {},\n 'f:configMap': {'.': {},\n 'f:defaultMode': {},\n 'f:name': {}},\n 'f:name': {}},\n 'k:{"name":"dshm"}': {'.': {},\n 'f:emptyDir': {'.': {},\n 'f:medium': {},\n [e2e-llm-inference-service] 'f:sizeLimit': {}},\n 'f:name': {}},\n 'k:{"name":"home"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"kserve-provision-location"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"model-cache"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"tls-certs"}': {'.': {},\n 'f:name': {},\n 'f:secret': {'.': {},\n 'f:defaultMode': {},\n 'f:secretName': {}}},\n 'k:{"name":"tmp-dir"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}}}}},\n 'manager': 'kube-controller-manager',\n 'operation': 'Update',\n 'subresource': None,\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.v1.cni.cncf.io/network-status': {}}}},\n 'manager': 'multus-daemon',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:status': {'f:conditions': {'k:{"type":"ContainersReady"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"Initialized"}': {'.': {},\n [e2e-llm-inference-service] 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodReadyToStartContainers"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodScheduled"}': {'f:observedGeneration': {}},\n 'k:{"type":"Ready"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}}},\n 'f:containerStatuses': {},\n 'f:hostIP': {},\n 'f:hostIPs': {},\n 'f:initContainerStatuses': {},\n 'f:observedGeneration': {},\n 'f:phase': {},\n 'f:podIP': {},\n 'f:podIPs': {'.': {},\n 'k:{"ip":"10.134.0.76"}': {'.': {},\n 'f:ip': {}}},\n 'f:startTime': {}}},\n 'manager': 'kubelet',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 51, 53, tzinfo=tzlocal())}],\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95-2tx5t',\n 'namespace': 'kserve-ci-e2e-test',\n 'owner_references': [{'api_version': 'apps/v1',\n 'block_owner_deletion': True,\n 'controller': True,\n 'kind': 'ReplicaSet',\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95',\n 'uid': '75c381c4-1742-4fb9-b05f-92ea6db35 [e2e-llm-inference-service] c9c'}],\n 'resource_version': '79769',\n 'self_link': None,\n 'uid': 'd4948f86-f0d1-4911-908f-7cb88db94468'},\n 'spec': {'active_deadline_seconds': None,\n 'affinity': None,\n 'automount_service_account_token': None,\n 'containers': [{'args': ['--enable-lora',\n '--max-lora-rank=64',\n '--max-loras=2',\n '--max-cpu-loras=4',\n '--lora-modules',\n '\'{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\''],\n 'command': ['/bin/bash',\n '-c',\n 'if [ -f /etc/profile.d/ibm-aiu-setup.sh '\n ']; then\n'\n ' source '\n '/etc/profile.d/ibm-aiu-setup.sh\n'\n 'fi\n'\n '\n'\n 'if [ "$KSERVE_INFER_ROCE" = "true" ]; '\n 'then\n'\n ' echo "Trying to infer RoCE configs '\n '... "\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gids/* '\n '2>/dev/null\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gid_attrs/types/* '\n '2>/dev/null\n'\n '\n'\n ' cat /proc/driver/nvidia/params\n'\n '\n'\n ' '\n 'KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE '\n 'v2"}\n'\n '\n'\n ' echo "[Infer RoCE] Discovering active '\n 'HCAs ..."\n'\n ' active_hcas=()\n'\n ' # Loop through all mlx5 devices found '\n 'in sysfs\n'\n ' for hca_dir in '\n '/sys/class/infiniband/mlx5_*; do\n'\n " # Ensure it's a directory before "\n 'proceeding\n'\n ' if [ -d "$hca_dir" ]; then\n'\n ' hca_name=$(basename '\n '"$hca_dir")\n'\n ' '\n 'port_state_file="$hca_dir/ports/1/state" '\n '# Assume port 1\n'\n ' '\n 'type_file="$hca_dir/ports/1/gid_attrs/types/*"\n'\n '\n'\n ' echo "[Infer RoCE] Check if '\n 'the port state file ${port_state_file} '\n 'exists and contains \'ACTIVE\'"\n'\n ' if [ -f "$port_state_file" ] '\n '&& grep -q "ACT [e2e-llm-inference-service] IVE" "$port_state_file" '\n '&& grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '${type_file} 2>/dev/null; then\n'\n ' echo "[Infer RoCE] Found '\n 'active HCA: $hca_name"\n'\n ' '\n 'active_hcas+=("$hca_name")\n'\n ' else\n'\n ' echo "[Infer RoCE] '\n 'Skipping inactive or down HCA: '\n '$hca_name"\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Check if we found any active HCAs\n'\n ' if [ ${#active_hcas[@]} -gt 0 ]; '\n 'then\n'\n ' # Join the array elements with a '\n 'comma\n'\n ' hca_port_pairs=()\n'\n ' for hca in "${active_hcas[@]}"; '\n 'do\n'\n ' hca_port_pairs+=("${hca}:1")\n'\n ' done\n'\n '\n'\n ' active_hca_list=$(IFS=,; echo '\n '"${active_hcas[*]}")\n'\n ' hca_port_pairs_list=$(IFS=,; echo '\n '"${hca_port_pairs[*]}")\n'\n ' echo "[Infer RoCE] Setting active '\n 'HCAs: ${active_hca_list}"\n'\n ' export '\n 'NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n'\n ' export '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n'\n ' export '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'NCCL_IB_HCA=${NCCL_IB_HCA}"\n'\n ' echo "[Infer RoCE] '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n'\n ' echo "[Infer RoCE] '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n'\n ' else\n'\n ' echo "[Infer RoCE] WARNING: No '\n 'active RoCE HCAs found. NCCL_IB_HCA '\n 'will not be set."\n'\n ' fi\n'\n '\n'\n ' if [ ${#active_hcas[@]} -gt 0 ]; '\n 'then\n'\n ' echo "[Infer RoCE] Finding '\n 'GID_INDEX for each active HCA (SR-IOV '\n 'compatible)..."\n'\n '\n'\n ' # For SR-IOV environments, find '\n 'the most common IPv4 RoCE v2 GID index '\n 'across all HCAs\n'\n ' declare -A gid_index_count\n'\n [e2e-llm-inference-service] ' declare -A hca_gid_index\n'\n '\n'\n ' for hca_name in '\n '"${active_hcas[@]}"; do\n'\n ' echo "[Infer RoCE] Processing '\n 'HCA: ${hca_name}"\n'\n '\n'\n ' # Find all RoCE v2 IPv4 GIDs '\n 'for this HCA and count by index\n'\n ' for tpath in '\n '/sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; '\n 'do\n'\n ' if grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '"$tpath" 2>/dev/null; then\n'\n ' idx=$(basename '\n '"$tpath")\n'\n ' '\n 'gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n'\n ' # Check for IPv4 GID '\n '(contains ffff:)\n'\n ' if [ -f "$gid_file" ] '\n '&& grep -q "ffff:" "$gid_file"; then\n'\n ' gid_value=$(cat '\n '"$gid_file" 2>/dev/null || echo "")\n'\n ' echo "[Infer '\n 'RoCE] Found IPv4 RoCE v2 GID for '\n '${hca_name}: index=${idx}, '\n 'gid=${gid_value}"\n'\n ' '\n 'hca_gid_index["${hca_name}"]="${idx}"\n'\n ' '\n 'gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} '\n '+ 1))\n'\n ' break # Use '\n 'first found IPv4 GID per HCA\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n ' done\n'\n '\n'\n ' # Find the most common GID index '\n '(most likely to be consistent across '\n 'nodes)\n'\n ' best_gid_index=""\n'\n ' max_count=0\n'\n ' for idx in '\n '"${!gid_index_count[@]}"; do\n'\n ' '\n 'count=${gid_index_count["${idx}"]}\n'\n ' echo "[Infer RoCE] GID_INDEX '\n '${idx} found on ${count} HCAs"\n'\n ' if [ $count -gt $max_count ]; '\n 'then\n'\n ' max_count=$count\n'\n ' best_gid_index="$idx"\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Use determinis [e2e-llm-inference-service] tic fallback if '\n 'tied - prefer index 3 (SR-IOV '\n 'standard)\n'\n ' if [ ${#gid_index_count[@]} -gt 1 '\n ']; then\n'\n ' echo "[Infer RoCE] Multiple '\n 'GID indices found, selecting most '\n 'common: ${best_gid_index}"\n'\n " # If there's a tie, prefer "\n "index 3 as it's most common in SR-IOV "\n 'setups\n'\n ' if [ -n '\n '"${gid_index_count[\'3\']}" ] && [ '\n '"${gid_index_count[\'3\']}" -eq '\n '"$max_count" ]; then\n'\n ' best_gid_index="3"\n'\n ' echo "[Infer RoCE] Using '\n 'deterministic fallback: GID_INDEX=3 '\n '(SR-IOV standard)"\n'\n ' fi\n'\n ' fi\n'\n '\n'\n ' # Check if GID_INDEX is already '\n 'set via environment variables\n'\n ' if [ -n "${NCCL_IB_GID_INDEX}" ]; '\n 'then\n'\n ' echo "[Infer RoCE] Using '\n 'pre-configured '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} '\n 'from environment"\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' echo "[Infer RoCE] Using '\n 'pre-configured '\n 'GID_INDEX=${NCCL_IB_GID_INDEX} for '\n 'NCCL, NVSHMEM, and UCX"\n'\n ' elif [ -n "$best_gid_index" ]; '\n 'then\n'\n ' echo "[Infer RoCE] Selected '\n 'GID_INDEX: ${best_gid_index} (found on '\n '${max_count} HCAs)"\n'\n '\n'\n ' export '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n'\n '\n'\n ' echo "[Infer RoCE] Exported '\n 'GID_INDEX=${best_gid_index} for NCCL, '\n 'NVSHMEM, and UCX"\n'\n ' else\n'\n ' echo "[Infer RoCE] ERROR: No '\n 'valid IPv4 '\n '${KSERVE_INFER_IB_GID_INDEX_GREP} '\n 'GID_INDEX found on any HCA."\n'\n [e2e-llm-inference-service] ' fi\n'\n ' else\n'\n ' echo "[Infer RoCE] No active HCAs '\n 'found, skipping GID_INDEX inference."\n'\n ' fi\n'\n 'fi\n'\n '\n'\n '# --disable-access-log-for-endpoints '\n 'landed in vLLM 0.16.0 '\n '(vllm-project/vllm#30011).\n'\n '# Older versions still need the blanket '\n '--disable-uvicorn-access-log.\n'\n 'ACCESS_LOG_ARGS="--disable-uvicorn-access-log"\n'\n 'VLLM_VERSION=$(vllm --version '\n "2>/dev/null | tail -1 | awk '{print "\n "$NF}')\n"\n 'echo "[access-log-detect] vllm '\n 'version=\'${VLLM_VERSION}\'"\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ "$(printf '\n '\'%s\\n%s\\n\' "0.16.0" '\n '"${VLLM_VERSION}" | sort -V | head -1)" '\n '= "0.16.0" ]; then\n'\n ' '\n 'ACCESS_LOG_ARGS="--disable-access-log-for-endpoints '\n '/health,/metrics,/ping"\n'\n 'fi\n'\n 'echo "[access-log-detect] selected '\n 'ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n'\n '\n'\n '# --shutdown-timeout landed in vLLM '\n '0.18.0 (vllm-project/vllm#36666).\n'\n 'SHUTDOWN_TIMEOUT_ARGS=""\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ "$(printf '\n '\'%s\\n%s\\n\' "0.18.0" '\n '"${VLLM_VERSION}" | sort -V | head -1)" '\n '= "0.18.0" ]; then\n'\n ' '\n 'SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout '\n '40"\n'\n 'fi\n'\n '\n'\n 'eval "exec vllm serve /mnt/models \\\n'\n ' --served-model-name '\n '"facebook/opt-125m" '\n '"publishers/kserve-ci-e2e-test/models/facebook/opt-125m" '\n '\\\n'\n ' --port 8000 \\\n'\n ' ${ACCESS_LOG_ARGS} \\\n'\n ' ${SHUTDOWN_TIMEOUT_ARGS} \\\n'\n ' --enable-ssl-refresh \\\n'\n ' --ssl-certfile '\n '/var/run/kserve/tls/tls.crt \\\n'\n ' --ssl-keyfile '\n '/var/run/kserve/tls/tls.key \\\n'\n ' ${VLLM_ADDITIONAL_ARGS} \\\n'\n ' $@"',\n '--'],\n 'env': [{'name': 'HOME',\n 'value': '/home',\n 'value_from': None},\n {'name': 'VLLM [e2e-llm-inference-service] _LOGGING_LEVEL',\n 'value': 'DEBUG',\n 'value_from': None},\n {'name': 'VLLM_CPU_KVCACHE_SPACE',\n 'value': '1',\n 'value_from': None},\n {'name': 'VLLM_ENABLE_V1_MULTIPROCESSING',\n 'value': '0',\n 'value_from': None},\n {'name': 'USER',\n 'value': 'nonroot',\n 'value_from': None},\n {'name': 'TORCHINDUCTOR_CACHE_DIR',\n 'value': '/tmp/torchinductor-cache',\n 'value_from': None},\n {'name': 'HF_HUB_CACHE',\n 'value': '/models',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': {'post_start': None,\n 'pre_stop': {'_exec': {'command': ['/bin/sleep',\n '15']},\n 'http_get': None,\n 'sleep': None,\n 'tcp_socket': None}},\n 'liveness_probe': {'_exec': None,\n 'failure_threshold': 10,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'name': 'main',\n 'ports': [{'container_port': 8000,\n 'host_ip': None,\n 'host_port': None,\n 'name': None,\n 'protocol': 'TCP'}],\n 'readiness_probe': {'_exec': None,\n 'failure_threshold': 2,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 1,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'resize_policy': [e2e-llm-inference-service] None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2', 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n 'read_only_root_filesystem': True,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'windows_options': None},\n 'startup_probe': {'_exec': None,\n 'failure_threshold': 60,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/home',\n 'mount_propagation': None,\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/tmp',\n 'mount_propagation': None,\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/dev/shm',\n 'mount_propagation': None,\n 'name': 'dshm',\n 'read_only': None,\n 'recursive_read_only': None,\n [e2e-llm-inference-service] 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/models',\n 'mount_propagation': None,\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/kserve/tls',\n 'mount_propagation': None,\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'dns_config': None,\n 'dns_policy': 'ClusterFirst',\n 'enable_service_links': True,\n 'ephemeral_containers': None,\n 'host_aliases': None,\n 'host_ipc': None,\n 'host_network': None,\n 'host_pid': None,\n 'host_users': None,\n 'hostname': None,\n 'image_pull_secrets': [{'name': 'default-dockercfg-6qcsh'}],\n 'init_containers': [{'args': ['hf://facebook/opt-125m',\n '/mnt/models',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-1',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-2'],\n 'command': None,\n 'env': [{'name': 'AWS_ACCESS_KEY_ID',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': None,\n 'secret_key_ref': {'key': 'AWS_ACCESS_KEY_ID',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'AWS_SECRET_ACCESS_KEY',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': N [e2e-llm-inference-service] one,\n 'secret_key_ref': {'key': 'AWS_SECRET_ACCESS_KEY',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'S3_USE_HTTPS',\n 'value': '0',\n 'value_from': None},\n {'name': 'S3_ENDPOINT',\n 'value': 's3-service.kserve:8333',\n 'value_from': None},\n {'name': 'AWS_ENDPOINT_URL',\n 'value': 'http://s3-service.kserve:8333',\n 'value_from': None},\n {'name': 'S3_VERIFY_SSL',\n 'value': '0',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE',\n 'value': '/etc/ssl/custom-certs/cabundle.crt',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE_CONFIGMAP',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'HF_HUB_ENABLE_HF_TRANSFER',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_HIGH_PERFORMANCE',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_NUM_CONCURRENT_RANGE_GETS',\n 'value': '8',\n 'value_from': None},\n {'name': 'CA_BUNDLE_CONFIGMAP_NAME',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'CA_BUNDLE_VOLUME_MOUNT_POINT',\n 'value': '/etc/ssl/custom-certs',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': None,\n 'liveness_probe': None,\n 'name': 'storage-initializer',\n 'ports': None,\n 'readiness_probe': None,\n 'resize_policy': None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n [e2e-llm-inference-service] 'read_only_root_filesystem': None,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': None,\n 'windows_options': None},\n 'startup_probe': None,\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'mount_propagation': None,\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'node_name': 'ip-10-0-131-57.ec2.internal',\n 'node_selector': None,\n 'os': None,\n 'overhead': None,\n 'preemption_policy': 'PreemptLowerPriority',\n 'priority': 0,\n 'priority_class_name': None,\n 'readiness_gates': None,\n 'resource_claims': None,\n 'resources': None,\n 'restart_policy': 'Always',\n 'runtime_class_name': None,\n 'scheduler_name': 'default-scheduler',\n 'scheduling_gates': None,\n 'security_context': {'app_armor_profile': None,\n 'fs_group': 1000690000,\n 'fs_group_change_policy': None,\n 'run_as_group': None,\n 'run_as_non_root': None,\n 'run_as_user': None,\n 'se_linux_change_policy': None,\n 'se_linux_options': {'level': 's0:c26,c20',\n 'role': None,\n 'type': None,\n 'user': None},\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'supplemental_groups': None,\n [e2e-llm-inference-service] 'supplemental_groups_policy': None,\n 'sysctls': None,\n 'windows_options': None},\n 'service_account': 'default',\n 'service_account_name': 'default',\n 'set_hostname_as_fqdn': None,\n 'share_process_namespace': None,\n 'subdomain': None,\n 'termination_grace_period_seconds': 60,\n 'tolerations': [{'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/not-ready',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/unreachable',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoSchedule',\n 'key': 'node.kubernetes.io/memory-pressure',\n 'operator': 'Exists',\n 'toleration_seconds': None,\n 'value': None}],\n 'topology_spread_constraints': None,\n 'volumes': [{'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'home',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': 'Memory', 'size_limit': '1Gi'},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'dshm',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_b [e2e-llm-inference-service] lock_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'model-cache',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tmp-dir',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tls-certs',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': {'default_mode': 420,\n 'items': None,\n [e2e-llm-inference-service] 'optional': None,\n 'secret_name': 'lora-multiple-adapters-test-kserve-self-signed-certs'},\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kserve-provision-location',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': {'default_mode': 420,\n 'items': None,\n 'name': 'odh-kserve-custom-ca-bundle',\n 'optional': None},\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'cabundle-cert',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kube-api-access-n5dws',\n [e2e-llm-inference-service] 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': {'default_mode': 420,\n 'sources': [{'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': {'audience': None,\n 'expiration_seconds': 3607,\n 'path': 'token'}},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'ca.crt',\n 'mode': None,\n 'path': 'ca.crt'}],\n 'name': 'kube-root-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': {'items': [{'field_ref': {'api_version': 'v1',\n 'field_path': 'metadata.namespace'},\n 'mode': None,\n 'path': 'namespace',\n 'resource_field_ref': None}]},\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'service-ca.crt',\n 'mode': None,\n 'path': 'service-ca.crt'}],\n 'name': 'openshift-service-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None}]},\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None}]},\n 'status': {'conditions': [{'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 18, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodReadyToStartContainers'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal()),\n 'message': None,\n [e2e-llm-inference-service] 'reason': None,\n 'status': 'True',\n 'type': 'Initialized'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'Ready'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'ContainersReady'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodScheduled'}],\n 'container_statuses': [{'allocated_resources': {'cpu': '200m',\n 'memory': '2Gi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402',\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_id': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'main',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2',\n 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_count': 0,\n 'started': True,\n 'state': {'running': {'started_at': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal())},\n 'terminated': None,\n 'waiting': None},\n 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/home',\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/tmp',\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/dev/shm',\n 'name': 'dshm',\n 'read_only': None,\n [e2e-llm-inference-service] 'recursive_read_only': None},\n {'mount_path': '/models',\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/var/run/kserve/tls',\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'ephemeral_container_statuses': None,\n 'host_i_ps': [{'ip': '10.0.131.57'}],\n 'host_ip': '10.0.131.57',\n 'init_container_statuses': [{'allocated_resources': {'cpu': '100m',\n 'memory': '100Mi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_id': 'quay.io/opendatahub/kserve-storage-initializer@sha256:60e2a5c8616d82c8933b614ae003dd3412a420a29a0be570b6a6439a3d39ab99',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'storage-initializer',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_count': 0,\n 'started': False,\n 'state': {'running': None,\n 'terminated': {'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'exit_code': 0,\n 'finished_at': datetime.datetime(2026, 7, 6, 21, 35, 22, tzinfo=tzlocal()),\n 'message': None,\n 'reason': 'Completed',\n 'signal': None,\n 'started_at': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n [e2e-llm-inference-service] 'waiting': None},\n 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'message': None,\n 'nominated_node_name': None,\n 'phase': 'Running',\n 'pod_i_ps': [{'ip': '10.134.0.76'}],\n 'pod_ip': '10.134.0.76',\n 'qos_class': 'Burstable',\n 'reason': None,\n 'resize': None,\n 'resource_claim_statuses': None,\n 'start_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())}}] = {'api_version': 'v1',\n 'items': [{'api_version': None,\n 'kind': None,\n 'metadata': {'annotations': {'k8s.ovn.org/pod-networks': '{"default":{"ip_addresses":["10.134.0.76/23"],"mac_address":"0a:58:0a:86:00:4c","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.76/23","gateway_ip":"10.134.0.1","role":"primary"}}',\n 'k8s.v1.cni.cncf.io/network-status': '[{\n'\n ' '\n '"name": '\n '"ovn-kubernetes",\n'\n ' '\n '"interface": '\n '"eth0",\n'\n ' '\n '"ips": '\n '[\n'\n ' '\n '"10.134.0.76"\n'\n ' '\n '],\n'\n ' '\n '"mac": '\n '"0a:58:0a:86:00:4c",\n'\n ' '\n [e2e-llm-inference-service] '"default": '\n 'true,\n'\n ' '\n '"dns": '\n '{}\n'\n '}]',\n 'openshift.io/scc': 'restricted-v2',\n 'seccomp.security.alpha.kubernetes.io/pod': 'runtime/default',\n 'security.openshift.io/validated-scc-subject-type': 'user'},\n 'creation_timestamp': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'deletion_grace_period_seconds': 60,\n 'deletion_timestamp': datetime.datetime(2026, 7, 6, 21, 52, 53, tzinfo=tzlocal()),\n 'finalizers': None,\n 'generate_name': 'lora-multiple-adapters-test-kserve-77485bff95-',\n 'generation': 2,\n 'labels': {'app.kubernetes.io/component': 'llminferenceservice-workload',\n 'app.kubernetes.io/name': 'lora-multiple-adapters-test',\n 'app.kubernetes.io/part-of': 'llminferenceservice',\n 'kserve.io/component': 'workload',\n 'llm-d.ai/role': 'both',\n 'pod-template-hash': '77485bff95'},\n 'managed_fields': [{'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.ovn.org/pod-networks': {}}}},\n 'manager': 'ip-10-0-131-57',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:generateName': {},\n 'f:labels': {'.': {},\n 'f:app.kubernetes.io/component': {},\n 'f:app.kubernetes.io/name': {},\n 'f:app.kubernetes.io/part-of': {},\n 'f:kserve.io/component': {},\n 'f:llm-d.ai/role': {},\n 'f:pod-template-hash': {}},\n 'f:ownerReferences': {'.': {},\n 'k:{"uid":"75c381c4-1742-4fb9-b05f-92ea6db35c9c"}': {}}},\n 'f:spec': {'f:containers': {'k:{"name":"main"}': {'.': {},\n 'f:args': {},\n 'f:command': {},\n [e2e-llm-inference-service] 'f:env': {'.': {},\n 'k:{"name":"HF_HUB_CACHE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HOME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"TORCHINDUCTOR_CACHE_DIR"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"USER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_CPU_KVCACHE_SPACE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_LOGGING_LEVEL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:lifecycle': {'.': {},\n 'f:pr [e2e-llm-inference-service] eStop': {'.': {},\n 'f:exec': {'.': {},\n 'f:command': {}}}},\n 'f:livenessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:name': {},\n 'f:ports': {'.': {},\n 'k:{"containerPort":8000,"protocol":"TCP"}': {'.': {},\n 'f:containerPort': {},\n 'f:protocol': {}}},\n 'f:readinessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:resources': {'.': {},\n [e2e-llm-inference-service] 'f:limits': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:securityContext': {'.': {},\n 'f:allowPrivilegeEscalation': {},\n 'f:capabilities': {'.': {},\n 'f:drop': {}},\n 'f:readOnlyRootFilesystem': {},\n 'f:runAsNonRoot': {},\n 'f:seccompProfile': {'.': {},\n 'f:type': {}}},\n 'f:startupProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:terminationMessagePath': {},\n 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n [e2e-llm-inference-service] 'k:{"mountPath":"/dev/shm"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/home"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/models"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/tmp"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/var/run/kserve/tls"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}}}}},\n 'f:dnsPolicy': {},\n 'f:enableServiceLinks': {},\n 'f:initContainers': {'.': {},\n 'k:{"name":"storage-initializer"}': {'.': {},\n 'f:args': {},\n [e2e-llm-inference-service] 'f:env': {'.': {},\n 'k:{"name":"AWS_ACCESS_KEY_ID"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"AWS_CA_BUNDLE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_ENDPOINT_URL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_SECRET_ACCESS_KEY"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}': {'.': {},\n [e2e-llm-inference-service] 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_HIGH_PERFORMANCE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_ENDPOINT"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_USE_HTTPS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_VERIFY_SSL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n [e2e-llm-inference-service] 'f:name': {},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:terminationMessagePath': {},\n 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/etc/ssl/custom-certs"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}}}}},\n 'f:restartPolicy': {},\n 'f:schedulerName': {},\n 'f:securityContext': {},\n 'f:terminationGracePeriodSeconds': {},\n 'f:volumes': {'.': {},\n 'k:{"name":"cabundle-cert"}': {'.': {},\n 'f:configMap': {'.': {},\n 'f:defaultMode': {} [e2e-llm-inference-service] ,\n 'f:name': {}},\n 'f:name': {}},\n 'k:{"name":"dshm"}': {'.': {},\n 'f:emptyDir': {'.': {},\n 'f:medium': {},\n 'f:sizeLimit': {}},\n 'f:name': {}},\n 'k:{"name":"home"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"kserve-provision-location"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"model-cache"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"tls-certs"}': {'.': {},\n 'f:name': {},\n 'f:secret': {'.': {},\n 'f:defaultMode': {},\n 'f:secretName': {}}},\n 'k:{"name":"tmp-dir"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}}}}},\n 'manager': 'kube-controller-manager',\n 'operation': 'Update',\n 'subresource': None,\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.v1.cni.cncf.io/network-status': {}}}},\n 'manager': 'multus-daemon',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\ [e2e-llm-inference-service] n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:status': {'f:conditions': {'k:{"type":"ContainersReady"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"Initialized"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodReadyToStartContainers"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodScheduled"}': {'f:observedGeneration': {}},\n 'k:{"type":"Ready"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}}},\n 'f:containerStatuses': {},\n 'f:hostIP': {},\n 'f:hostIPs': {},\n 'f:initContainerStatuses': {},\n [e2e-llm-inference-service] 'f:observedGeneration': {},\n 'f:phase': {},\n 'f:podIP': {},\n 'f:podIPs': {'.': {},\n 'k:{"ip":"10.134.0.76"}': {'.': {},\n 'f:ip': {}}},\n 'f:startTime': {}}},\n 'manager': 'kubelet',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 51, 53, tzinfo=tzlocal())}],\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95-2tx5t',\n 'namespace': 'kserve-ci-e2e-test',\n 'owner_references': [{'api_version': 'apps/v1',\n 'block_owner_deletion': True,\n 'controller': True,\n 'kind': 'ReplicaSet',\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95',\n 'uid': '75c381c4-1742-4fb9-b05f-92ea6db35c9c'}],\n 'resource_version': '79769',\n 'self_link': None,\n 'uid': 'd4948f86-f0d1-4911-908f-7cb88db94468'},\n 'spec': {'active_deadline_seconds': None,\n 'affinity': None,\n 'automount_service_account_token': None,\n 'containers': [{'args': ['--enable-lora',\n '--max-lora-rank=64',\n '--max-loras=2',\n '--max-cpu-loras=4',\n '--lora-modules',\n '\'{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\''],\n 'command': ['/bin/bash',\n '-c',\n 'if [ -f '\n '/etc/profile.d/ibm-aiu-setup.sh '\n ']; then\n'\n ' source '\n '/etc/profile.d/ibm-aiu-setup.sh\n'\n 'fi\n'\n '\n'\n 'if [ "$KSERVE_INFER_ROCE" = '\n '"true" ]; then\n'\n ' echo "Trying to infer RoCE '\n 'configs ... "\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gids/* '\n '2>/dev/null\n'\n ' grep -H . '\n '/sys/class/inf [e2e-llm-inference-service] iniband/*/ports/*/gid_attrs/types/* '\n '2>/dev/null\n'\n '\n'\n ' cat '\n '/proc/driver/nvidia/params\n'\n '\n'\n ' '\n 'KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE '\n 'v2"}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'Discovering active HCAs '\n '..."\n'\n ' active_hcas=()\n'\n ' # Loop through all mlx5 '\n 'devices found in sysfs\n'\n ' for hca_dir in '\n '/sys/class/infiniband/mlx5_*; '\n 'do\n'\n " # Ensure it's a "\n 'directory before proceeding\n'\n ' if [ -d "$hca_dir" ]; '\n 'then\n'\n ' '\n 'hca_name=$(basename '\n '"$hca_dir")\n'\n ' '\n 'port_state_file="$hca_dir/ports/1/state" '\n '# Assume port 1\n'\n ' '\n 'type_file="$hca_dir/ports/1/gid_attrs/types/*"\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'Check if the port state file '\n '${port_state_file} exists '\n 'and contains \'ACTIVE\'"\n'\n ' if [ -f '\n '"$port_state_file" ] && grep '\n '-q "ACTIVE" '\n '"$port_state_file" && grep '\n '-q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '${type_file} 2>/dev/null; '\n 'then\n'\n ' echo "[Infer '\n 'RoCE] Found active HCA: '\n '$hca_name"\n'\n ' '\n 'active_hcas+=("$hca_name")\n'\n ' else\n'\n ' echo "[Infer '\n 'RoCE] Skipping inactive or '\n 'down HCA: $hca_name"\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n '\n'\n [e2e-llm-inference-service] ' # Check if we found any '\n 'active HCAs\n'\n ' if [ ${#active_hcas[@]} '\n '-gt 0 ]; then\n'\n ' # Join the array '\n 'elements with a comma\n'\n ' hca_port_pairs=()\n'\n ' for hca in '\n '"${active_hcas[@]}"; do\n'\n ' '\n 'hca_port_pairs+=("${hca}:1")\n'\n ' done\n'\n '\n'\n ' '\n 'active_hca_list=$(IFS=,; '\n 'echo "${active_hcas[*]}")\n'\n ' '\n 'hca_port_pairs_list=$(IFS=,; '\n 'echo '\n '"${hca_port_pairs[*]}")\n'\n ' echo "[Infer RoCE] '\n 'Setting active HCAs: '\n '${active_hca_list}"\n'\n ' export '\n 'NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n'\n ' export '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n'\n ' export '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'NCCL_IB_HCA=${NCCL_IB_HCA}"\n'\n ' echo "[Infer RoCE] '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n'\n ' echo "[Infer RoCE] '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n'\n ' else\n'\n ' echo "[Infer RoCE] '\n 'WARNING: No active RoCE HCAs '\n 'found. NCCL_IB_HCA will not '\n 'be set."\n'\n ' fi\n'\n '\n'\n ' if [ ${#active_hcas[@]} '\n '-gt 0 ]; then\n'\n ' echo "[Infer RoCE] '\n 'Finding GID_INDEX for each '\n 'active HCA (SR-IOV '\n 'compatible)..."\n'\n '\n'\n ' # For SR-IOV '\n 'environments, find the most '\n 'common IPv4 RoCE v2 GID '\n 'index across all HCAs\n'\n ' declare -A '\n [e2e-llm-inference-service] 'gid_index_count\n'\n ' declare -A '\n 'hca_gid_index\n'\n '\n'\n ' for hca_name in '\n '"${active_hcas[@]}"; do\n'\n ' echo "[Infer RoCE] '\n 'Processing HCA: '\n '${hca_name}"\n'\n '\n'\n ' # Find all RoCE v2 '\n 'IPv4 GIDs for this HCA and '\n 'count by index\n'\n ' for tpath in '\n '/sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; '\n 'do\n'\n ' if grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '"$tpath" 2>/dev/null; then\n'\n ' '\n 'idx=$(basename "$tpath")\n'\n ' '\n 'gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n'\n ' # Check '\n 'for IPv4 GID (contains '\n 'ffff:)\n'\n ' if [ -f '\n '"$gid_file" ] && grep -q '\n '"ffff:" "$gid_file"; then\n'\n ' '\n 'gid_value=$(cat "$gid_file" '\n '2>/dev/null || echo "")\n'\n ' echo '\n '"[Infer RoCE] Found IPv4 '\n 'RoCE v2 GID for ${hca_name}: '\n 'index=${idx}, '\n 'gid=${gid_value}"\n'\n ' '\n 'hca_gid_index["${hca_name}"]="${idx}"\n'\n ' '\n 'gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} '\n '+ 1))\n'\n ' break '\n '# Use first found IPv4 GID '\n 'per HCA\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n ' done\n'\n '\n'\n ' # Find the most common '\n 'GID index (most likely to be '\n 'consistent across nodes)\n'\n ' best_gid_index=""\n'\n [e2e-llm-inference-service] ' max_count=0\n'\n ' for idx in '\n '"${!gid_index_count[@]}"; '\n 'do\n'\n ' '\n 'count=${gid_index_count["${idx}"]}\n'\n ' echo "[Infer RoCE] '\n 'GID_INDEX ${idx} found on '\n '${count} HCAs"\n'\n ' if [ $count -gt '\n '$max_count ]; then\n'\n ' '\n 'max_count=$count\n'\n ' '\n 'best_gid_index="$idx"\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Use deterministic '\n 'fallback if tied - prefer '\n 'index 3 (SR-IOV standard)\n'\n ' if [ '\n '${#gid_index_count[@]} -gt 1 '\n ']; then\n'\n ' echo "[Infer RoCE] '\n 'Multiple GID indices found, '\n 'selecting most common: '\n '${best_gid_index}"\n'\n " # If there's a "\n "tie, prefer index 3 as it's "\n 'most common in SR-IOV '\n 'setups\n'\n ' if [ -n '\n '"${gid_index_count[\'3\']}" '\n '] && [ '\n '"${gid_index_count[\'3\']}" '\n '-eq "$max_count" ]; then\n'\n ' '\n 'best_gid_index="3"\n'\n ' echo "[Infer '\n 'RoCE] Using deterministic '\n 'fallback: GID_INDEX=3 '\n '(SR-IOV standard)"\n'\n ' fi\n'\n ' fi\n'\n '\n'\n ' # Check if GID_INDEX '\n 'is already set via '\n 'environment variables\n'\n ' if [ -n '\n '"${NCCL_IB_GID_INDEX}" ]; '\n 'then\n'\n ' echo "[Infer RoCE] '\n 'Using pre-configured '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} '\n 'from environment"\n'\n [e2e-llm-inference-service] ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' echo "[Infer RoCE] '\n 'Using pre-configured '\n 'GID_INDEX=${NCCL_IB_GID_INDEX} '\n 'for NCCL, NVSHMEM, and UCX"\n'\n ' elif [ -n '\n '"$best_gid_index" ]; then\n'\n ' echo "[Infer RoCE] '\n 'Selected GID_INDEX: '\n '${best_gid_index} (found on '\n '${max_count} HCAs)"\n'\n '\n'\n ' export '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'Exported '\n 'GID_INDEX=${best_gid_index} '\n 'for NCCL, NVSHMEM, and UCX"\n'\n ' else\n'\n ' echo "[Infer RoCE] '\n 'ERROR: No valid IPv4 '\n '${KSERVE_INFER_IB_GID_INDEX_GREP} '\n 'GID_INDEX found on any '\n 'HCA."\n'\n ' fi\n'\n ' else\n'\n ' echo "[Infer RoCE] No '\n 'active HCAs found, skipping '\n 'GID_INDEX inference."\n'\n ' fi\n'\n 'fi\n'\n '\n'\n '# '\n '--disable-access-log-for-endpoints '\n 'landed in vLLM 0.16.0 '\n '(vllm-project/vllm#30011).\n'\n '# Older versions still need '\n 'the blanket '\n '--disable-uvicorn-access-log.\n'\n 'ACCESS_LOG_ARGS="--disable-uvicorn-access-log"\n'\n 'VLLM_VERSION=$(vllm '\n '--version 2>/dev/null | tail '\n "-1 | awk '{print $NF}')\n"\n 'echo "[access-log-detect] '\n 'vllm '\n [e2e-llm-inference-service] 'version=\'${VLLM_VERSION}\'"\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ '\n '"$(printf \'%s\\n%s\\n\' '\n '"0.16.0" "${VLLM_VERSION}" | '\n 'sort -V | head -1)" = '\n '"0.16.0" ]; then\n'\n ' '\n 'ACCESS_LOG_ARGS="--disable-access-log-for-endpoints '\n '/health,/metrics,/ping"\n'\n 'fi\n'\n 'echo "[access-log-detect] '\n 'selected '\n 'ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n'\n '\n'\n '# --shutdown-timeout landed '\n 'in vLLM 0.18.0 '\n '(vllm-project/vllm#36666).\n'\n 'SHUTDOWN_TIMEOUT_ARGS=""\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ '\n '"$(printf \'%s\\n%s\\n\' '\n '"0.18.0" "${VLLM_VERSION}" | '\n 'sort -V | head -1)" = '\n '"0.18.0" ]; then\n'\n ' '\n 'SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout '\n '40"\n'\n 'fi\n'\n '\n'\n 'eval "exec vllm serve '\n '/mnt/models \\\n'\n ' --served-model-name '\n '"facebook/opt-125m" '\n '"publishers/kserve-ci-e2e-test/models/facebook/opt-125m" '\n '\\\n'\n ' --port 8000 \\\n'\n ' ${ACCESS_LOG_ARGS} \\\n'\n ' ${SHUTDOWN_TIMEOUT_ARGS} '\n '\\\n'\n ' --enable-ssl-refresh \\\n'\n ' --ssl-certfile '\n '/var/run/kserve/tls/tls.crt '\n '\\\n'\n ' --ssl-keyfile '\n '/var/run/kserve/tls/tls.key '\n '\\\n'\n ' ${VLLM_ADDITIONAL_ARGS} '\n '\\\n'\n ' $@"',\n '--'],\n 'env': [{'name': 'HOME',\n 'value': '/home',\n 'value_from': None},\n {'name': 'VLLM_LOGGING_LEVEL',\n 'value': 'DEBUG',\n 'value_from': None},\n [e2e-llm-inference-service] {'name': 'VLLM_CPU_KVCACHE_SPACE',\n 'value': '1',\n 'value_from': None},\n {'name': 'VLLM_ENABLE_V1_MULTIPROCESSING',\n 'value': '0',\n 'value_from': None},\n {'name': 'USER',\n 'value': 'nonroot',\n 'value_from': None},\n {'name': 'TORCHINDUCTOR_CACHE_DIR',\n 'value': '/tmp/torchinductor-cache',\n 'value_from': None},\n {'name': 'HF_HUB_CACHE',\n 'value': '/models',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': {'post_start': None,\n 'pre_stop': {'_exec': {'command': ['/bin/sleep',\n '15']},\n 'http_get': None,\n 'sleep': None,\n 'tcp_socket': None}},\n 'liveness_probe': {'_exec': None,\n 'failure_threshold': 10,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'name': 'main',\n 'ports': [{'container_port': 8000,\n 'host_ip': None,\n 'host_port': None,\n 'name': None,\n 'protocol': 'TCP'}],\n 'readiness_probe': {'_exec': None,\n 'failure_threshold': 2,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n [e2e-llm-inference-service] 'initial_delay_seconds': None,\n 'period_seconds': 1,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'resize_policy': None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2',\n 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n 'read_only_root_filesystem': True,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'windows_options': None},\n 'startup_probe': {'_exec': None,\n 'failure_threshold': 60,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/home',\n 'mount_propagation': None,\n 'name': 'home',\n 'read_only': None,\n [e2e-llm-inference-service] 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/tmp',\n 'mount_propagation': None,\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/dev/shm',\n 'mount_propagation': None,\n 'name': 'dshm',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/models',\n 'mount_propagation': None,\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/kserve/tls',\n 'mount_propagation': None,\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'dns_config': None,\n 'dns_policy': 'ClusterFirst',\n 'enable_service_links': True,\n 'ephemeral_containers': None,\n 'host_aliases': None,\n 'host_ipc': None,\n 'host_network': None,\n 'host_pid' [e2e-llm-inference-service] : None,\n 'host_users': None,\n 'hostname': None,\n 'image_pull_secrets': [{'name': 'default-dockercfg-6qcsh'}],\n 'init_containers': [{'args': ['hf://facebook/opt-125m',\n '/mnt/models',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-1',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-2'],\n 'command': None,\n 'env': [{'name': 'AWS_ACCESS_KEY_ID',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': None,\n 'secret_key_ref': {'key': 'AWS_ACCESS_KEY_ID',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'AWS_SECRET_ACCESS_KEY',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': None,\n 'secret_key_ref': {'key': 'AWS_SECRET_ACCESS_KEY',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'S3_USE_HTTPS',\n 'value': '0',\n 'value_from': None},\n {'name': 'S3_ENDPOINT',\n 'value': 's3-service.kserve:8333',\n 'value_from': None},\n {'name': 'AWS_ENDPOINT_URL',\n 'value': 'http://s3-service.kserve:8333',\n 'value_from': None},\n {'name': 'S3_VERIFY_SSL',\n 'value': '0',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE',\n 'value': '/etc/ssl/custom-certs/cabundle.crt',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE_CONFIGMAP',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'HF_HUB_ENABLE_HF_TRANSFER',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_HIGH_PERFORMANCE',\n 'value': '1',\n 'value_fro [e2e-llm-inference-service] m': None},\n {'name': 'HF_XET_NUM_CONCURRENT_RANGE_GETS',\n 'value': '8',\n 'value_from': None},\n {'name': 'CA_BUNDLE_CONFIGMAP_NAME',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'CA_BUNDLE_VOLUME_MOUNT_POINT',\n 'value': '/etc/ssl/custom-certs',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': None,\n 'liveness_probe': None,\n 'name': 'storage-initializer',\n 'ports': None,\n 'readiness_probe': None,\n 'resize_policy': None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n 'read_only_root_filesystem': None,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': None,\n 'windows_options': None},\n 'startup_probe': None,\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None,\n [e2e-llm-inference-service] 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'mount_propagation': None,\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'node_name': 'ip-10-0-131-57.ec2.internal',\n 'node_selector': None,\n 'os': None,\n 'overhead': None,\n 'preemption_policy': 'PreemptLowerPriority',\n 'priority': 0,\n 'priority_class_name': None,\n 'readiness_gates': None,\n 'resource_claims': None,\n 'resources': None,\n 'restart_policy': 'Always',\n 'runtime_class_name': None,\n 'scheduler_name': 'default-scheduler',\n 'scheduling_gates': None,\n 'security_context': {'app_armor_profile': None,\n 'fs_group': 1000690000,\n 'fs_group_change_policy': None,\n 'run_as_group': None,\n 'run_as_non_root': None,\n 'run_as_user': None,\n 'se_linux_change_policy': None,\n 'se_linux_options': {'level': 's0:c26,c20',\n 'role': None,\n 'type': None,\n 'user': None},\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'supplemental_groups': None,\n 'supplemental_groups_policy': None,\n 'sysctls': None,\n 'windows_options': None},\n 'service_account': 'default',\n 'service_account_name': 'default',\n 'set_hostname_as_fqdn': None,\n 'share_process_namespace': None,\n 'subdomain': None,\n 'termination_grace_period_seconds': 60,\n 'tolerations': [{'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/not-ready',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n [e2e-llm-inference-service] {'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/unreachable',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoSchedule',\n 'key': 'node.kubernetes.io/memory-pressure',\n 'operator': 'Exists',\n 'toleration_seconds': None,\n 'value': None}],\n 'topology_spread_constraints': None,\n 'volumes': [{'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None,\n 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'home',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': 'Memory',\n 'size_limit': '1Gi'},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'dshm',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n [e2e-llm-inference-service] 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None,\n 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'model-cache',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None,\n 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tmp-dir',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n [e2e-llm-inference-service] 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tls-certs',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': {'default_mode': 420,\n 'items': None,\n 'optional': None,\n 'secret_name': 'lora-multiple-adapters-test-kserve-self-signed-certs'},\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None,\n 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kserve-provision-location',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': {'default_mode': 420,\n 'items': None,\n 'name': 'odh-kserve-custom-ca-bundle',\n 'optional': None},\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\ [e2e-llm-inference-service] n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'cabundle-cert',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kube-api-access-n5dws',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': {'default_mode': 420,\n 'sources': [{'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': {'audience': None,\n 'expiration_seconds': 3607,\n 'path': 'token'}},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'ca.crt',\n 'mode': None,\n 'path': 'ca.crt'}],\n 'name': 'kube-root-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n [e2e-llm-inference-service] 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': {'items': [{'field_ref': {'api_version': 'v1',\n 'field_path': 'metadata.namespace'},\n 'mode': None,\n 'path': 'namespace',\n 'resource_field_ref': None}]},\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'service-ca.crt',\n 'mode': None,\n 'path': 'service-ca.crt'}],\n 'name': 'openshift-service-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None}]},\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None}]},\n 'status': {'conditions': [{'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 18, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodReadyToStartContainers'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'Initialized'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'Ready'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'ContainersReady'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzin [e2e-llm-inference-service] fo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodScheduled'}],\n 'container_statuses': [{'allocated_resources': {'cpu': '200m',\n 'memory': '2Gi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402',\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_id': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'main',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2',\n 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_count': 0,\n 'started': True,\n 'state': {'running': {'started_at': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal())},\n 'terminated': None,\n 'waiting': None},\n 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/home',\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/tmp',\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/dev/shm',\n 'name': 'dshm',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/models',\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/var/run/kserve/tls',\n [e2e-llm-inference-service] 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'ephemeral_container_statuses': None,\n 'host_i_ps': [{'ip': '10.0.131.57'}],\n 'host_ip': '10.0.131.57',\n 'init_container_statuses': [{'allocated_resources': {'cpu': '100m',\n 'memory': '100Mi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_id': 'quay.io/opendatahub/kserve-storage-initializer@sha256:60e2a5c8616d82c8933b614ae003dd3412a420a29a0be570b6a6439a3d39ab99',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'storage-initializer',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_count': 0,\n 'started': False,\n 'state': {'running': None,\n 'terminated': {'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'exit_code': 0,\n 'finished_at': datetime.datetime(2026, 7, 6, 21, 35, 22, tzinfo=tzlocal()),\n 'message': None,\n 'reason': 'Completed',\n 'signal': None,\n 'started_at': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n 'waiting': None},\n [e2e-llm-inference-service] 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'message': None,\n 'nominated_node_name': None,\n 'phase': 'Running',\n 'pod_i_ps': [{'ip': '10.134.0.76'}],\n 'pod_ip': '10.134.0.76',\n 'qos_class': 'Burstable',\n 'reason': None,\n 'resize': None,\n 'resource_claim_statuses': None,\n 'start_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())}}],\n 'kind': 'PodList',\n 'metadata': {'_continue': None,\n 'remaining_item_count': None,\n 'resource_version': '79871',\n 'self_link': None}}.items [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: 1 pod(s) for lora-multiple-adapters-test still terminating [e2e-llm-inference-service] assert not [{'api_version': None,\n 'kind': None,\n 'metadata': {'annotations': {'k8s.ovn.org/pod-networks': '{"default":{"ip_addresses":["10.134.0.76/23"],"mac_address":"0a:58:0a:86:00:4c","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.76/23","gateway_ip":"10.134.0.1","role":"primary"}}',\n 'k8s.v1.cni.cncf.io/network-status': '[{\n'\n ' '\n '"name": '\n '"ovn-kubernetes",\n'\n ' '\n '"interface": '\n '"eth0",\n'\n ' "ips": '\n '[\n'\n ' '\n '"10.134.0.76"\n'\n ' ],\n'\n ' "mac": '\n '"0a:58:0a:86:00:4c",\n'\n ' '\n '"default": '\n 'true,\n'\n ' "dns": '\n '{}\n'\n '}]',\n 'openshift.io/scc': 'restricted-v2',\n 'seccomp.security.alpha.kubernetes.io/pod': 'runtime/default',\n 'security.openshift.io/validated-scc-subject-type': 'user'},\n 'creation_timestamp': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'deletion_grace_period_seconds': 60,\n 'deletion_timestamp': datetime.datetime(2026, 7, 6, 21, 52, 53, tzinfo=tzlocal()),\n 'finalizers': None,\n 'generate_name': 'lora-multiple-adapters-test-kserve-77485bff95-',\n 'generation': 2,\n 'labels': {'app.kubernetes.io/component': 'llminferenceservice-workload',\n 'app.kubernetes.io/name': 'lora-multiple-adapters-test',\n 'app.kubernetes.io/part-of': 'llminferenceservice',\n 'kserve.io/component': 'workload',\n 'llm-d.ai/role': 'both',\n 'pod-template-hash': '77485bff95'},\n 'managed_fields': [{'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.ovn.org/pod-networks': {}}}},\n 'manager': 'ip-10-0-131-57',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:generateName': {},\n 'f:labels': {'.': {},\n 'f:app.kubernetes.io/component' [e2e-llm-inference-service] : {},\n 'f:app.kubernetes.io/name': {},\n 'f:app.kubernetes.io/part-of': {},\n 'f:kserve.io/component': {},\n 'f:llm-d.ai/role': {},\n 'f:pod-template-hash': {}},\n 'f:ownerReferences': {'.': {},\n 'k:{"uid":"75c381c4-1742-4fb9-b05f-92ea6db35c9c"}': {}}},\n 'f:spec': {'f:containers': {'k:{"name":"main"}': {'.': {},\n 'f:args': {},\n 'f:command': {},\n 'f:env': {'.': {},\n 'k:{"name":"HF_HUB_CACHE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HOME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"TORCHINDUCTOR_CACHE_DIR"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"USER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_CPU_KVCACHE_SPACE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n [e2e-llm-inference-service] 'k:{"name":"VLLM_LOGGING_LEVEL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:lifecycle': {'.': {},\n 'f:preStop': {'.': {},\n 'f:exec': {'.': {},\n 'f:command': {}}}},\n 'f:livenessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:name': {},\n 'f:ports': {'.': {},\n 'k:{"containerPort":8000,"protocol":"TCP"}': {'.': {},\n 'f:containerPort': {},\n 'f:protocol': {}}},\n 'f:readinessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n [e2e-llm-inference-service] 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:securityContext': {'.': {},\n 'f:allowPrivilegeEscalation': {},\n 'f:capabilities': {'.': {},\n 'f:drop': {}},\n 'f:readOnlyRootFilesystem': {},\n 'f:runAsNonRoot': {},\n 'f:seccompProfile': {'.': {},\n 'f:type': {}}},\n 'f:startupProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:terminationMessagePath': {},\n [e2e-llm-inference-service] 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/dev/shm"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/home"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/models"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/tmp"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/var/run/kserve/tls"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}}}}},\n 'f:dnsPolicy': {},\n 'f:enableServiceLinks': {},\n 'f:initContainers': {'.': {},\n 'k:{"name":"storage-initializer"}': {'.': {},\n 'f:args': {},\n [e2e-llm-inference-service] 'f:env': {'.': {},\n 'k:{"name":"AWS_ACCESS_KEY_ID"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"AWS_CA_BUNDLE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_ENDPOINT_URL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_SECRET_ACCESS_KEY"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}': {'.': {},\n 'f:name': {},\n [e2e-llm-inference-service] 'f:value': {}},\n 'k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_HIGH_PERFORMANCE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_ENDPOINT"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_USE_HTTPS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_VERIFY_SSL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:name': {},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n [e2e-llm-inference-service] 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:terminationMessagePath': {},\n 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/etc/ssl/custom-certs"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}}}}},\n 'f:restartPolicy': {},\n 'f:schedulerName': {},\n 'f:securityContext': {},\n 'f:terminationGracePeriodSeconds': {},\n 'f:volumes': {'.': {},\n 'k:{"name":"cabundle-cert"}': {'.': {},\n 'f:configMap': {'.': {},\n 'f:defaultMode': {},\n 'f:name': {}},\n 'f:name': {}},\n 'k:{"name":"dshm"}': {'.': {},\n 'f:emptyDir': {'.': {},\n 'f:medium': {},\n [e2e-llm-inference-service] 'f:sizeLimit': {}},\n 'f:name': {}},\n 'k:{"name":"home"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"kserve-provision-location"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"model-cache"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"tls-certs"}': {'.': {},\n 'f:name': {},\n 'f:secret': {'.': {},\n 'f:defaultMode': {},\n 'f:secretName': {}}},\n 'k:{"name":"tmp-dir"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}}}}},\n 'manager': 'kube-controller-manager',\n 'operation': 'Update',\n 'subresource': None,\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.v1.cni.cncf.io/network-status': {}}}},\n 'manager': 'multus-daemon',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:status': {'f:conditions': {'k:{"type":"ContainersReady"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"Initialized"}': {'.': {},\n [e2e-llm-inference-service] 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodReadyToStartContainers"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodScheduled"}': {'f:observedGeneration': {}},\n 'k:{"type":"Ready"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}}},\n 'f:containerStatuses': {},\n 'f:hostIP': {},\n 'f:hostIPs': {},\n 'f:initContainerStatuses': {},\n 'f:observedGeneration': {},\n 'f:phase': {},\n 'f:podIP': {},\n 'f:podIPs': {'.': {},\n 'k:{"ip":"10.134.0.76"}': {'.': {},\n 'f:ip': {}}},\n 'f:startTime': {}}},\n 'manager': 'kubelet',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 51, 53, tzinfo=tzlocal())}],\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95-2tx5t',\n 'namespace': 'kserve-ci-e2e-test',\n 'owner_references': [{'api_version': 'apps/v1',\n 'block_owner_deletion': True,\n 'controller': True,\n 'kind': 'ReplicaSet',\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95',\n 'uid': '75c381c4-1742-4fb9-b05f-92ea6db3 [e2e-llm-inference-service] 5c9c'}],\n 'resource_version': '79769',\n 'self_link': None,\n 'uid': 'd4948f86-f0d1-4911-908f-7cb88db94468'},\n 'spec': {'active_deadline_seconds': None,\n 'affinity': None,\n 'automount_service_account_token': None,\n 'containers': [{'args': ['--enable-lora',\n '--max-lora-rank=64',\n '--max-loras=2',\n '--max-cpu-loras=4',\n '--lora-modules',\n '\'{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\''],\n 'command': ['/bin/bash',\n '-c',\n 'if [ -f /etc/profile.d/ibm-aiu-setup.sh '\n ']; then\n'\n ' source '\n '/etc/profile.d/ibm-aiu-setup.sh\n'\n 'fi\n'\n '\n'\n 'if [ "$KSERVE_INFER_ROCE" = "true" ]; '\n 'then\n'\n ' echo "Trying to infer RoCE configs '\n '... "\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gids/* '\n '2>/dev/null\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gid_attrs/types/* '\n '2>/dev/null\n'\n '\n'\n ' cat /proc/driver/nvidia/params\n'\n '\n'\n ' '\n 'KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE '\n 'v2"}\n'\n '\n'\n ' echo "[Infer RoCE] Discovering active '\n 'HCAs ..."\n'\n ' active_hcas=()\n'\n ' # Loop through all mlx5 devices found '\n 'in sysfs\n'\n ' for hca_dir in '\n '/sys/class/infiniband/mlx5_*; do\n'\n " # Ensure it's a directory before "\n 'proceeding\n'\n ' if [ -d "$hca_dir" ]; then\n'\n ' hca_name=$(basename '\n '"$hca_dir")\n'\n ' '\n 'port_state_file="$hca_dir/ports/1/state" '\n '# Assume port 1\n'\n ' '\n 'type_file="$hca_dir/ports/1/gid_attrs/types/*"\n'\n '\n'\n ' echo "[Infer RoCE] Check if '\n 'the port state file ${port_state_file} '\n 'exists and contains \'ACTIVE\'"\n'\n ' if [ -f "$port_state_file" ] '\n '&& grep -q "AC [e2e-llm-inference-service] TIVE" "$port_state_file" '\n '&& grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '${type_file} 2>/dev/null; then\n'\n ' echo "[Infer RoCE] Found '\n 'active HCA: $hca_name"\n'\n ' '\n 'active_hcas+=("$hca_name")\n'\n ' else\n'\n ' echo "[Infer RoCE] '\n 'Skipping inactive or down HCA: '\n '$hca_name"\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Check if we found any active HCAs\n'\n ' if [ ${#active_hcas[@]} -gt 0 ]; '\n 'then\n'\n ' # Join the array elements with a '\n 'comma\n'\n ' hca_port_pairs=()\n'\n ' for hca in "${active_hcas[@]}"; '\n 'do\n'\n ' hca_port_pairs+=("${hca}:1")\n'\n ' done\n'\n '\n'\n ' active_hca_list=$(IFS=,; echo '\n '"${active_hcas[*]}")\n'\n ' hca_port_pairs_list=$(IFS=,; echo '\n '"${hca_port_pairs[*]}")\n'\n ' echo "[Infer RoCE] Setting active '\n 'HCAs: ${active_hca_list}"\n'\n ' export '\n 'NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n'\n ' export '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n'\n ' export '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'NCCL_IB_HCA=${NCCL_IB_HCA}"\n'\n ' echo "[Infer RoCE] '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n'\n ' echo "[Infer RoCE] '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n'\n ' else\n'\n ' echo "[Infer RoCE] WARNING: No '\n 'active RoCE HCAs found. NCCL_IB_HCA '\n 'will not be set."\n'\n ' fi\n'\n '\n'\n ' if [ ${#active_hcas[@]} -gt 0 ]; '\n 'then\n'\n ' echo "[Infer RoCE] Finding '\n 'GID_INDEX for each active HCA (SR-IOV '\n 'compatible)..."\n'\n '\n'\n ' # For SR-IOV environments, find '\n 'the most common IPv4 RoCE v2 GID index '\n 'across all HCAs\n'\n ' declare -A gid_index_count\n'\n [e2e-llm-inference-service] ' declare -A hca_gid_index\n'\n '\n'\n ' for hca_name in '\n '"${active_hcas[@]}"; do\n'\n ' echo "[Infer RoCE] Processing '\n 'HCA: ${hca_name}"\n'\n '\n'\n ' # Find all RoCE v2 IPv4 GIDs '\n 'for this HCA and count by index\n'\n ' for tpath in '\n '/sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; '\n 'do\n'\n ' if grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '"$tpath" 2>/dev/null; then\n'\n ' idx=$(basename '\n '"$tpath")\n'\n ' '\n 'gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n'\n ' # Check for IPv4 GID '\n '(contains ffff:)\n'\n ' if [ -f "$gid_file" ] '\n '&& grep -q "ffff:" "$gid_file"; then\n'\n ' gid_value=$(cat '\n '"$gid_file" 2>/dev/null || echo "")\n'\n ' echo "[Infer '\n 'RoCE] Found IPv4 RoCE v2 GID for '\n '${hca_name}: index=${idx}, '\n 'gid=${gid_value}"\n'\n ' '\n 'hca_gid_index["${hca_name}"]="${idx}"\n'\n ' '\n 'gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} '\n '+ 1))\n'\n ' break # Use '\n 'first found IPv4 GID per HCA\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n ' done\n'\n '\n'\n ' # Find the most common GID index '\n '(most likely to be consistent across '\n 'nodes)\n'\n ' best_gid_index=""\n'\n ' max_count=0\n'\n ' for idx in '\n '"${!gid_index_count[@]}"; do\n'\n ' '\n 'count=${gid_index_count["${idx}"]}\n'\n ' echo "[Infer RoCE] GID_INDEX '\n '${idx} found on ${count} HCAs"\n'\n ' if [ $count -gt $max_count ]; '\n 'then\n'\n ' max_count=$count\n'\n ' best_gid_index="$idx"\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Use determini [e2e-llm-inference-service] stic fallback if '\n 'tied - prefer index 3 (SR-IOV '\n 'standard)\n'\n ' if [ ${#gid_index_count[@]} -gt 1 '\n ']; then\n'\n ' echo "[Infer RoCE] Multiple '\n 'GID indices found, selecting most '\n 'common: ${best_gid_index}"\n'\n " # If there's a tie, prefer "\n "index 3 as it's most common in SR-IOV "\n 'setups\n'\n ' if [ -n '\n '"${gid_index_count[\'3\']}" ] && [ '\n '"${gid_index_count[\'3\']}" -eq '\n '"$max_count" ]; then\n'\n ' best_gid_index="3"\n'\n ' echo "[Infer RoCE] Using '\n 'deterministic fallback: GID_INDEX=3 '\n '(SR-IOV standard)"\n'\n ' fi\n'\n ' fi\n'\n '\n'\n ' # Check if GID_INDEX is already '\n 'set via environment variables\n'\n ' if [ -n "${NCCL_IB_GID_INDEX}" ]; '\n 'then\n'\n ' echo "[Infer RoCE] Using '\n 'pre-configured '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} '\n 'from environment"\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' echo "[Infer RoCE] Using '\n 'pre-configured '\n 'GID_INDEX=${NCCL_IB_GID_INDEX} for '\n 'NCCL, NVSHMEM, and UCX"\n'\n ' elif [ -n "$best_gid_index" ]; '\n 'then\n'\n ' echo "[Infer RoCE] Selected '\n 'GID_INDEX: ${best_gid_index} (found on '\n '${max_count} HCAs)"\n'\n '\n'\n ' export '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n'\n '\n'\n ' echo "[Infer RoCE] Exported '\n 'GID_INDEX=${best_gid_index} for NCCL, '\n 'NVSHMEM, and UCX"\n'\n ' else\n'\n ' echo "[Infer RoCE] ERROR: No '\n 'valid IPv4 '\n '${KSERVE_INFER_IB_GID_INDEX_GREP} '\n 'GID_INDEX found on any HCA."\n'\n [e2e-llm-inference-service] ' fi\n'\n ' else\n'\n ' echo "[Infer RoCE] No active HCAs '\n 'found, skipping GID_INDEX inference."\n'\n ' fi\n'\n 'fi\n'\n '\n'\n '# --disable-access-log-for-endpoints '\n 'landed in vLLM 0.16.0 '\n '(vllm-project/vllm#30011).\n'\n '# Older versions still need the blanket '\n '--disable-uvicorn-access-log.\n'\n 'ACCESS_LOG_ARGS="--disable-uvicorn-access-log"\n'\n 'VLLM_VERSION=$(vllm --version '\n "2>/dev/null | tail -1 | awk '{print "\n "$NF}')\n"\n 'echo "[access-log-detect] vllm '\n 'version=\'${VLLM_VERSION}\'"\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ "$(printf '\n '\'%s\\n%s\\n\' "0.16.0" '\n '"${VLLM_VERSION}" | sort -V | head -1)" '\n '= "0.16.0" ]; then\n'\n ' '\n 'ACCESS_LOG_ARGS="--disable-access-log-for-endpoints '\n '/health,/metrics,/ping"\n'\n 'fi\n'\n 'echo "[access-log-detect] selected '\n 'ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n'\n '\n'\n '# --shutdown-timeout landed in vLLM '\n '0.18.0 (vllm-project/vllm#36666).\n'\n 'SHUTDOWN_TIMEOUT_ARGS=""\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ "$(printf '\n '\'%s\\n%s\\n\' "0.18.0" '\n '"${VLLM_VERSION}" | sort -V | head -1)" '\n '= "0.18.0" ]; then\n'\n ' '\n 'SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout '\n '40"\n'\n 'fi\n'\n '\n'\n 'eval "exec vllm serve /mnt/models \\\n'\n ' --served-model-name '\n '"facebook/opt-125m" '\n '"publishers/kserve-ci-e2e-test/models/facebook/opt-125m" '\n '\\\n'\n ' --port 8000 \\\n'\n ' ${ACCESS_LOG_ARGS} \\\n'\n ' ${SHUTDOWN_TIMEOUT_ARGS} \\\n'\n ' --enable-ssl-refresh \\\n'\n ' --ssl-certfile '\n '/var/run/kserve/tls/tls.crt \\\n'\n ' --ssl-keyfile '\n '/var/run/kserve/tls/tls.key \\\n'\n ' ${VLLM_ADDITIONAL_ARGS} \\\n'\n ' $@"',\n '--'],\n 'env': [{'name': 'HOME',\n 'value': '/home',\n 'value_from': None},\n {'name': 'VLL [e2e-llm-inference-service] M_LOGGING_LEVEL',\n 'value': 'DEBUG',\n 'value_from': None},\n {'name': 'VLLM_CPU_KVCACHE_SPACE',\n 'value': '1',\n 'value_from': None},\n {'name': 'VLLM_ENABLE_V1_MULTIPROCESSING',\n 'value': '0',\n 'value_from': None},\n {'name': 'USER',\n 'value': 'nonroot',\n 'value_from': None},\n {'name': 'TORCHINDUCTOR_CACHE_DIR',\n 'value': '/tmp/torchinductor-cache',\n 'value_from': None},\n {'name': 'HF_HUB_CACHE',\n 'value': '/models',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': {'post_start': None,\n 'pre_stop': {'_exec': {'command': ['/bin/sleep',\n '15']},\n 'http_get': None,\n 'sleep': None,\n 'tcp_socket': None}},\n 'liveness_probe': {'_exec': None,\n 'failure_threshold': 10,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'name': 'main',\n 'ports': [{'container_port': 8000,\n 'host_ip': None,\n 'host_port': None,\n 'name': None,\n 'protocol': 'TCP'}],\n 'readiness_probe': {'_exec': None,\n 'failure_threshold': 2,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 1,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'resize_policy': [e2e-llm-inference-service] None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2', 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n 'read_only_root_filesystem': True,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'windows_options': None},\n 'startup_probe': {'_exec': None,\n 'failure_threshold': 60,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/home',\n 'mount_propagation': None,\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/tmp',\n 'mount_propagation': None,\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/dev/shm',\n 'mount_propagation': None,\n 'name': 'dshm',\n 'read_only': None,\n 'recursive_read_only': None,\n [e2e-llm-inference-service] 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/models',\n 'mount_propagation': None,\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/kserve/tls',\n 'mount_propagation': None,\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'dns_config': None,\n 'dns_policy': 'ClusterFirst',\n 'enable_service_links': True,\n 'ephemeral_containers': None,\n 'host_aliases': None,\n 'host_ipc': None,\n 'host_network': None,\n 'host_pid': None,\n 'host_users': None,\n 'hostname': None,\n 'image_pull_secrets': [{'name': 'default-dockercfg-6qcsh'}],\n 'init_containers': [{'args': ['hf://facebook/opt-125m',\n '/mnt/models',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-1',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-2'],\n 'command': None,\n 'env': [{'name': 'AWS_ACCESS_KEY_ID',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': None,\n 'secret_key_ref': {'key': 'AWS_ACCESS_KEY_ID',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'AWS_SECRET_ACCESS_KEY',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': [e2e-llm-inference-service] None,\n 'secret_key_ref': {'key': 'AWS_SECRET_ACCESS_KEY',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'S3_USE_HTTPS',\n 'value': '0',\n 'value_from': None},\n {'name': 'S3_ENDPOINT',\n 'value': 's3-service.kserve:8333',\n 'value_from': None},\n {'name': 'AWS_ENDPOINT_URL',\n 'value': 'http://s3-service.kserve:8333',\n 'value_from': None},\n {'name': 'S3_VERIFY_SSL',\n 'value': '0',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE',\n 'value': '/etc/ssl/custom-certs/cabundle.crt',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE_CONFIGMAP',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'HF_HUB_ENABLE_HF_TRANSFER',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_HIGH_PERFORMANCE',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_NUM_CONCURRENT_RANGE_GETS',\n 'value': '8',\n 'value_from': None},\n {'name': 'CA_BUNDLE_CONFIGMAP_NAME',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'CA_BUNDLE_VOLUME_MOUNT_POINT',\n 'value': '/etc/ssl/custom-certs',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': None,\n 'liveness_probe': None,\n 'name': 'storage-initializer',\n 'ports': None,\n 'readiness_probe': None,\n 'resize_policy': None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n [e2e-llm-inference-service] 'read_only_root_filesystem': None,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': None,\n 'windows_options': None},\n 'startup_probe': None,\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'mount_propagation': None,\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'node_name': 'ip-10-0-131-57.ec2.internal',\n 'node_selector': None,\n 'os': None,\n 'overhead': None,\n 'preemption_policy': 'PreemptLowerPriority',\n 'priority': 0,\n 'priority_class_name': None,\n 'readiness_gates': None,\n 'resource_claims': None,\n 'resources': None,\n 'restart_policy': 'Always',\n 'runtime_class_name': None,\n 'scheduler_name': 'default-scheduler',\n 'scheduling_gates': None,\n 'security_context': {'app_armor_profile': None,\n 'fs_group': 1000690000,\n 'fs_group_change_policy': None,\n 'run_as_group': None,\n 'run_as_non_root': None,\n 'run_as_user': None,\n 'se_linux_change_policy': None,\n 'se_linux_options': {'level': 's0:c26,c20',\n 'role': None,\n 'type': None,\n 'user': None},\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'supplemental_groups': None,\n [e2e-llm-inference-service] 'supplemental_groups_policy': None,\n 'sysctls': None,\n 'windows_options': None},\n 'service_account': 'default',\n 'service_account_name': 'default',\n 'set_hostname_as_fqdn': None,\n 'share_process_namespace': None,\n 'subdomain': None,\n 'termination_grace_period_seconds': 60,\n 'tolerations': [{'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/not-ready',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/unreachable',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoSchedule',\n 'key': 'node.kubernetes.io/memory-pressure',\n 'operator': 'Exists',\n 'toleration_seconds': None,\n 'value': None}],\n 'topology_spread_constraints': None,\n 'volumes': [{'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'home',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': 'Memory', 'size_limit': '1Gi'},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'dshm',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_ [e2e-llm-inference-service] block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'model-cache',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tmp-dir',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tls-certs',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': {'default_mode': 420,\n 'items': None,\n [e2e-llm-inference-service] 'optional': None,\n 'secret_name': 'lora-multiple-adapters-test-kserve-self-signed-certs'},\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kserve-provision-location',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': {'default_mode': 420,\n 'items': None,\n 'name': 'odh-kserve-custom-ca-bundle',\n 'optional': None},\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'cabundle-cert',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kube-api-access-n5dws',\n [e2e-llm-inference-service] 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': {'default_mode': 420,\n 'sources': [{'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': {'audience': None,\n 'expiration_seconds': 3607,\n 'path': 'token'}},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'ca.crt',\n 'mode': None,\n 'path': 'ca.crt'}],\n 'name': 'kube-root-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': {'items': [{'field_ref': {'api_version': 'v1',\n 'field_path': 'metadata.namespace'},\n 'mode': None,\n 'path': 'namespace',\n 'resource_field_ref': None}]},\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'service-ca.crt',\n 'mode': None,\n 'path': 'service-ca.crt'}],\n 'name': 'openshift-service-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None}]},\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None}]},\n 'status': {'conditions': [{'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 18, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodReadyToStartContainers'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal()),\n 'message': None,\n [e2e-llm-inference-service] 'reason': None,\n 'status': 'True',\n 'type': 'Initialized'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'Ready'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'ContainersReady'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodScheduled'}],\n 'container_statuses': [{'allocated_resources': {'cpu': '200m',\n 'memory': '2Gi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402',\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_id': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'main',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2',\n 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_count': 0,\n 'started': True,\n 'state': {'running': {'started_at': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal())},\n 'terminated': None,\n 'waiting': None},\n 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/home',\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/tmp',\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/dev/shm',\n 'name': 'dshm',\n 'read_only': None,\n [e2e-llm-inference-service] 'recursive_read_only': None},\n {'mount_path': '/models',\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/var/run/kserve/tls',\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'ephemeral_container_statuses': None,\n 'host_i_ps': [{'ip': '10.0.131.57'}],\n 'host_ip': '10.0.131.57',\n 'init_container_statuses': [{'allocated_resources': {'cpu': '100m',\n 'memory': '100Mi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_id': 'quay.io/opendatahub/kserve-storage-initializer@sha256:60e2a5c8616d82c8933b614ae003dd3412a420a29a0be570b6a6439a3d39ab99',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'storage-initializer',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_count': 0,\n 'started': False,\n 'state': {'running': None,\n 'terminated': {'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'exit_code': 0,\n 'finished_at': datetime.datetime(2026, 7, 6, 21, 35, 22, tzinfo=tzlocal()),\n 'message': None,\n 'reason': 'Completed',\n 'signal': None,\n 'started_at': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n [e2e-llm-inference-service] 'waiting': None},\n 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'message': None,\n 'nominated_node_name': None,\n 'phase': 'Running',\n 'pod_i_ps': [{'ip': '10.134.0.76'}],\n 'pod_ip': '10.134.0.76',\n 'qos_class': 'Burstable',\n 'reason': None,\n 'resize': None,\n 'resource_claim_statuses': None,\n 'start_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())}}] [e2e-llm-inference-service] + where [{'api_version': None,\n 'kind': None,\n 'metadata': {'annotations': {'k8s.ovn.org/pod-networks': '{"default":{"ip_addresses":["10.134.0.76/23"],"mac_address":"0a:58:0a:86:00:4c","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.76/23","gateway_ip":"10.134.0.1","role":"primary"}}',\n 'k8s.v1.cni.cncf.io/network-status': '[{\n'\n ' '\n '"name": '\n '"ovn-kubernetes",\n'\n ' '\n '"interface": '\n '"eth0",\n'\n ' "ips": '\n '[\n'\n ' '\n '"10.134.0.76"\n'\n ' ],\n'\n ' "mac": '\n '"0a:58:0a:86:00:4c",\n'\n ' '\n '"default": '\n 'true,\n'\n ' "dns": '\n '{}\n'\n '}]',\n 'openshift.io/scc': 'restricted-v2',\n 'seccomp.security.alpha.kubernetes.io/pod': 'runtime/default',\n 'security.openshift.io/validated-scc-subject-type': 'user'},\n 'creation_timestamp': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'deletion_grace_period_seconds': 60,\n 'deletion_timestamp': datetime.datetime(2026, 7, 6, 21, 52, 53, tzinfo=tzlocal()),\n 'finalizers': None,\n 'generate_name': 'lora-multiple-adapters-test-kserve-77485bff95-',\n 'generation': 2,\n 'labels': {'app.kubernetes.io/component': 'llminferenceservice-workload',\n 'app.kubernetes.io/name': 'lora-multiple-adapters-test',\n 'app.kubernetes.io/part-of': 'llminferenceservice',\n 'kserve.io/component': 'workload',\n 'llm-d.ai/role': 'both',\n 'pod-template-hash': '77485bff95'},\n 'managed_fields': [{'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.ovn.org/pod-networks': {}}}},\n 'manager': 'ip-10-0-131-57',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:generateName': {},\n 'f:labels': {'.': {},\n 'f:app.kubernetes.io/component': [e2e-llm-inference-service] {},\n 'f:app.kubernetes.io/name': {},\n 'f:app.kubernetes.io/part-of': {},\n 'f:kserve.io/component': {},\n 'f:llm-d.ai/role': {},\n 'f:pod-template-hash': {}},\n 'f:ownerReferences': {'.': {},\n 'k:{"uid":"75c381c4-1742-4fb9-b05f-92ea6db35c9c"}': {}}},\n 'f:spec': {'f:containers': {'k:{"name":"main"}': {'.': {},\n 'f:args': {},\n 'f:command': {},\n 'f:env': {'.': {},\n 'k:{"name":"HF_HUB_CACHE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HOME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"TORCHINDUCTOR_CACHE_DIR"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"USER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_CPU_KVCACHE_SPACE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n [e2e-llm-inference-service] 'k:{"name":"VLLM_LOGGING_LEVEL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:lifecycle': {'.': {},\n 'f:preStop': {'.': {},\n 'f:exec': {'.': {},\n 'f:command': {}}}},\n 'f:livenessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:name': {},\n 'f:ports': {'.': {},\n 'k:{"containerPort":8000,"protocol":"TCP"}': {'.': {},\n 'f:containerPort': {},\n 'f:protocol': {}}},\n 'f:readinessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n [e2e-llm-inference-service] 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:securityContext': {'.': {},\n 'f:allowPrivilegeEscalation': {},\n 'f:capabilities': {'.': {},\n 'f:drop': {}},\n 'f:readOnlyRootFilesystem': {},\n 'f:runAsNonRoot': {},\n 'f:seccompProfile': {'.': {},\n 'f:type': {}}},\n 'f:startupProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:terminationMessagePath': {},\n [e2e-llm-inference-service] 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/dev/shm"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/home"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/models"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/tmp"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/var/run/kserve/tls"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}}}}},\n 'f:dnsPolicy': {},\n 'f:enableServiceLinks': {},\n 'f:initContainers': {'.': {},\n 'k:{"name":"storage-initializer"}': {'.': {},\n 'f:args': {},\n [e2e-llm-inference-service] 'f:env': {'.': {},\n 'k:{"name":"AWS_ACCESS_KEY_ID"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"AWS_CA_BUNDLE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_ENDPOINT_URL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_SECRET_ACCESS_KEY"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}': {'.': {},\n 'f:name': {},\n [e2e-llm-inference-service] 'f:value': {}},\n 'k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_HIGH_PERFORMANCE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_ENDPOINT"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_USE_HTTPS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_VERIFY_SSL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:name': {},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n [e2e-llm-inference-service] 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:terminationMessagePath': {},\n 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/etc/ssl/custom-certs"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}}}}},\n 'f:restartPolicy': {},\n 'f:schedulerName': {},\n 'f:securityContext': {},\n 'f:terminationGracePeriodSeconds': {},\n 'f:volumes': {'.': {},\n 'k:{"name":"cabundle-cert"}': {'.': {},\n 'f:configMap': {'.': {},\n 'f:defaultMode': {},\n 'f:name': {}},\n 'f:name': {}},\n 'k:{"name":"dshm"}': {'.': {},\n 'f:emptyDir': {'.': {},\n 'f:medium': {},\n [e2e-llm-inference-service] 'f:sizeLimit': {}},\n 'f:name': {}},\n 'k:{"name":"home"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"kserve-provision-location"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"model-cache"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"tls-certs"}': {'.': {},\n 'f:name': {},\n 'f:secret': {'.': {},\n 'f:defaultMode': {},\n 'f:secretName': {}}},\n 'k:{"name":"tmp-dir"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}}}}},\n 'manager': 'kube-controller-manager',\n 'operation': 'Update',\n 'subresource': None,\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.v1.cni.cncf.io/network-status': {}}}},\n 'manager': 'multus-daemon',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:status': {'f:conditions': {'k:{"type":"ContainersReady"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"Initialized"}': {'.': {},\n [e2e-llm-inference-service] 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodReadyToStartContainers"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodScheduled"}': {'f:observedGeneration': {}},\n 'k:{"type":"Ready"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}}},\n 'f:containerStatuses': {},\n 'f:hostIP': {},\n 'f:hostIPs': {},\n 'f:initContainerStatuses': {},\n 'f:observedGeneration': {},\n 'f:phase': {},\n 'f:podIP': {},\n 'f:podIPs': {'.': {},\n 'k:{"ip":"10.134.0.76"}': {'.': {},\n 'f:ip': {}}},\n 'f:startTime': {}}},\n 'manager': 'kubelet',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 51, 53, tzinfo=tzlocal())}],\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95-2tx5t',\n 'namespace': 'kserve-ci-e2e-test',\n 'owner_references': [{'api_version': 'apps/v1',\n 'block_owner_deletion': True,\n 'controller': True,\n 'kind': 'ReplicaSet',\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95',\n 'uid': '75c381c4-1742-4fb9-b05f-92ea6db35 [e2e-llm-inference-service] c9c'}],\n 'resource_version': '79769',\n 'self_link': None,\n 'uid': 'd4948f86-f0d1-4911-908f-7cb88db94468'},\n 'spec': {'active_deadline_seconds': None,\n 'affinity': None,\n 'automount_service_account_token': None,\n 'containers': [{'args': ['--enable-lora',\n '--max-lora-rank=64',\n '--max-loras=2',\n '--max-cpu-loras=4',\n '--lora-modules',\n '\'{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\''],\n 'command': ['/bin/bash',\n '-c',\n 'if [ -f /etc/profile.d/ibm-aiu-setup.sh '\n ']; then\n'\n ' source '\n '/etc/profile.d/ibm-aiu-setup.sh\n'\n 'fi\n'\n '\n'\n 'if [ "$KSERVE_INFER_ROCE" = "true" ]; '\n 'then\n'\n ' echo "Trying to infer RoCE configs '\n '... "\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gids/* '\n '2>/dev/null\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gid_attrs/types/* '\n '2>/dev/null\n'\n '\n'\n ' cat /proc/driver/nvidia/params\n'\n '\n'\n ' '\n 'KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE '\n 'v2"}\n'\n '\n'\n ' echo "[Infer RoCE] Discovering active '\n 'HCAs ..."\n'\n ' active_hcas=()\n'\n ' # Loop through all mlx5 devices found '\n 'in sysfs\n'\n ' for hca_dir in '\n '/sys/class/infiniband/mlx5_*; do\n'\n " # Ensure it's a directory before "\n 'proceeding\n'\n ' if [ -d "$hca_dir" ]; then\n'\n ' hca_name=$(basename '\n '"$hca_dir")\n'\n ' '\n 'port_state_file="$hca_dir/ports/1/state" '\n '# Assume port 1\n'\n ' '\n 'type_file="$hca_dir/ports/1/gid_attrs/types/*"\n'\n '\n'\n ' echo "[Infer RoCE] Check if '\n 'the port state file ${port_state_file} '\n 'exists and contains \'ACTIVE\'"\n'\n ' if [ -f "$port_state_file" ] '\n '&& grep -q "ACT [e2e-llm-inference-service] IVE" "$port_state_file" '\n '&& grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '${type_file} 2>/dev/null; then\n'\n ' echo "[Infer RoCE] Found '\n 'active HCA: $hca_name"\n'\n ' '\n 'active_hcas+=("$hca_name")\n'\n ' else\n'\n ' echo "[Infer RoCE] '\n 'Skipping inactive or down HCA: '\n '$hca_name"\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Check if we found any active HCAs\n'\n ' if [ ${#active_hcas[@]} -gt 0 ]; '\n 'then\n'\n ' # Join the array elements with a '\n 'comma\n'\n ' hca_port_pairs=()\n'\n ' for hca in "${active_hcas[@]}"; '\n 'do\n'\n ' hca_port_pairs+=("${hca}:1")\n'\n ' done\n'\n '\n'\n ' active_hca_list=$(IFS=,; echo '\n '"${active_hcas[*]}")\n'\n ' hca_port_pairs_list=$(IFS=,; echo '\n '"${hca_port_pairs[*]}")\n'\n ' echo "[Infer RoCE] Setting active '\n 'HCAs: ${active_hca_list}"\n'\n ' export '\n 'NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n'\n ' export '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n'\n ' export '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'NCCL_IB_HCA=${NCCL_IB_HCA}"\n'\n ' echo "[Infer RoCE] '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n'\n ' echo "[Infer RoCE] '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n'\n ' else\n'\n ' echo "[Infer RoCE] WARNING: No '\n 'active RoCE HCAs found. NCCL_IB_HCA '\n 'will not be set."\n'\n ' fi\n'\n '\n'\n ' if [ ${#active_hcas[@]} -gt 0 ]; '\n 'then\n'\n ' echo "[Infer RoCE] Finding '\n 'GID_INDEX for each active HCA (SR-IOV '\n 'compatible)..."\n'\n '\n'\n ' # For SR-IOV environments, find '\n 'the most common IPv4 RoCE v2 GID index '\n 'across all HCAs\n'\n ' declare -A gid_index_count\n'\n [e2e-llm-inference-service] ' declare -A hca_gid_index\n'\n '\n'\n ' for hca_name in '\n '"${active_hcas[@]}"; do\n'\n ' echo "[Infer RoCE] Processing '\n 'HCA: ${hca_name}"\n'\n '\n'\n ' # Find all RoCE v2 IPv4 GIDs '\n 'for this HCA and count by index\n'\n ' for tpath in '\n '/sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; '\n 'do\n'\n ' if grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '"$tpath" 2>/dev/null; then\n'\n ' idx=$(basename '\n '"$tpath")\n'\n ' '\n 'gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n'\n ' # Check for IPv4 GID '\n '(contains ffff:)\n'\n ' if [ -f "$gid_file" ] '\n '&& grep -q "ffff:" "$gid_file"; then\n'\n ' gid_value=$(cat '\n '"$gid_file" 2>/dev/null || echo "")\n'\n ' echo "[Infer '\n 'RoCE] Found IPv4 RoCE v2 GID for '\n '${hca_name}: index=${idx}, '\n 'gid=${gid_value}"\n'\n ' '\n 'hca_gid_index["${hca_name}"]="${idx}"\n'\n ' '\n 'gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} '\n '+ 1))\n'\n ' break # Use '\n 'first found IPv4 GID per HCA\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n ' done\n'\n '\n'\n ' # Find the most common GID index '\n '(most likely to be consistent across '\n 'nodes)\n'\n ' best_gid_index=""\n'\n ' max_count=0\n'\n ' for idx in '\n '"${!gid_index_count[@]}"; do\n'\n ' '\n 'count=${gid_index_count["${idx}"]}\n'\n ' echo "[Infer RoCE] GID_INDEX '\n '${idx} found on ${count} HCAs"\n'\n ' if [ $count -gt $max_count ]; '\n 'then\n'\n ' max_count=$count\n'\n ' best_gid_index="$idx"\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Use determinis [e2e-llm-inference-service] tic fallback if '\n 'tied - prefer index 3 (SR-IOV '\n 'standard)\n'\n ' if [ ${#gid_index_count[@]} -gt 1 '\n ']; then\n'\n ' echo "[Infer RoCE] Multiple '\n 'GID indices found, selecting most '\n 'common: ${best_gid_index}"\n'\n " # If there's a tie, prefer "\n "index 3 as it's most common in SR-IOV "\n 'setups\n'\n ' if [ -n '\n '"${gid_index_count[\'3\']}" ] && [ '\n '"${gid_index_count[\'3\']}" -eq '\n '"$max_count" ]; then\n'\n ' best_gid_index="3"\n'\n ' echo "[Infer RoCE] Using '\n 'deterministic fallback: GID_INDEX=3 '\n '(SR-IOV standard)"\n'\n ' fi\n'\n ' fi\n'\n '\n'\n ' # Check if GID_INDEX is already '\n 'set via environment variables\n'\n ' if [ -n "${NCCL_IB_GID_INDEX}" ]; '\n 'then\n'\n ' echo "[Infer RoCE] Using '\n 'pre-configured '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} '\n 'from environment"\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' echo "[Infer RoCE] Using '\n 'pre-configured '\n 'GID_INDEX=${NCCL_IB_GID_INDEX} for '\n 'NCCL, NVSHMEM, and UCX"\n'\n ' elif [ -n "$best_gid_index" ]; '\n 'then\n'\n ' echo "[Infer RoCE] Selected '\n 'GID_INDEX: ${best_gid_index} (found on '\n '${max_count} HCAs)"\n'\n '\n'\n ' export '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n'\n '\n'\n ' echo "[Infer RoCE] Exported '\n 'GID_INDEX=${best_gid_index} for NCCL, '\n 'NVSHMEM, and UCX"\n'\n ' else\n'\n ' echo "[Infer RoCE] ERROR: No '\n 'valid IPv4 '\n '${KSERVE_INFER_IB_GID_INDEX_GREP} '\n 'GID_INDEX found on any HCA."\n'\n [e2e-llm-inference-service] ' fi\n'\n ' else\n'\n ' echo "[Infer RoCE] No active HCAs '\n 'found, skipping GID_INDEX inference."\n'\n ' fi\n'\n 'fi\n'\n '\n'\n '# --disable-access-log-for-endpoints '\n 'landed in vLLM 0.16.0 '\n '(vllm-project/vllm#30011).\n'\n '# Older versions still need the blanket '\n '--disable-uvicorn-access-log.\n'\n 'ACCESS_LOG_ARGS="--disable-uvicorn-access-log"\n'\n 'VLLM_VERSION=$(vllm --version '\n "2>/dev/null | tail -1 | awk '{print "\n "$NF}')\n"\n 'echo "[access-log-detect] vllm '\n 'version=\'${VLLM_VERSION}\'"\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ "$(printf '\n '\'%s\\n%s\\n\' "0.16.0" '\n '"${VLLM_VERSION}" | sort -V | head -1)" '\n '= "0.16.0" ]; then\n'\n ' '\n 'ACCESS_LOG_ARGS="--disable-access-log-for-endpoints '\n '/health,/metrics,/ping"\n'\n 'fi\n'\n 'echo "[access-log-detect] selected '\n 'ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n'\n '\n'\n '# --shutdown-timeout landed in vLLM '\n '0.18.0 (vllm-project/vllm#36666).\n'\n 'SHUTDOWN_TIMEOUT_ARGS=""\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ "$(printf '\n '\'%s\\n%s\\n\' "0.18.0" '\n '"${VLLM_VERSION}" | sort -V | head -1)" '\n '= "0.18.0" ]; then\n'\n ' '\n 'SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout '\n '40"\n'\n 'fi\n'\n '\n'\n 'eval "exec vllm serve /mnt/models \\\n'\n ' --served-model-name '\n '"facebook/opt-125m" '\n '"publishers/kserve-ci-e2e-test/models/facebook/opt-125m" '\n '\\\n'\n ' --port 8000 \\\n'\n ' ${ACCESS_LOG_ARGS} \\\n'\n ' ${SHUTDOWN_TIMEOUT_ARGS} \\\n'\n ' --enable-ssl-refresh \\\n'\n ' --ssl-certfile '\n '/var/run/kserve/tls/tls.crt \\\n'\n ' --ssl-keyfile '\n '/var/run/kserve/tls/tls.key \\\n'\n ' ${VLLM_ADDITIONAL_ARGS} \\\n'\n ' $@"',\n '--'],\n 'env': [{'name': 'HOME',\n 'value': '/home',\n 'value_from': None},\n {'name': 'VLLM [e2e-llm-inference-service] _LOGGING_LEVEL',\n 'value': 'DEBUG',\n 'value_from': None},\n {'name': 'VLLM_CPU_KVCACHE_SPACE',\n 'value': '1',\n 'value_from': None},\n {'name': 'VLLM_ENABLE_V1_MULTIPROCESSING',\n 'value': '0',\n 'value_from': None},\n {'name': 'USER',\n 'value': 'nonroot',\n 'value_from': None},\n {'name': 'TORCHINDUCTOR_CACHE_DIR',\n 'value': '/tmp/torchinductor-cache',\n 'value_from': None},\n {'name': 'HF_HUB_CACHE',\n 'value': '/models',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': {'post_start': None,\n 'pre_stop': {'_exec': {'command': ['/bin/sleep',\n '15']},\n 'http_get': None,\n 'sleep': None,\n 'tcp_socket': None}},\n 'liveness_probe': {'_exec': None,\n 'failure_threshold': 10,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'name': 'main',\n 'ports': [{'container_port': 8000,\n 'host_ip': None,\n 'host_port': None,\n 'name': None,\n 'protocol': 'TCP'}],\n 'readiness_probe': {'_exec': None,\n 'failure_threshold': 2,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 1,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'resize_policy': [e2e-llm-inference-service] None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2', 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n 'read_only_root_filesystem': True,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'windows_options': None},\n 'startup_probe': {'_exec': None,\n 'failure_threshold': 60,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/home',\n 'mount_propagation': None,\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/tmp',\n 'mount_propagation': None,\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/dev/shm',\n 'mount_propagation': None,\n 'name': 'dshm',\n 'read_only': None,\n 'recursive_read_only': None,\n [e2e-llm-inference-service] 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/models',\n 'mount_propagation': None,\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/kserve/tls',\n 'mount_propagation': None,\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'dns_config': None,\n 'dns_policy': 'ClusterFirst',\n 'enable_service_links': True,\n 'ephemeral_containers': None,\n 'host_aliases': None,\n 'host_ipc': None,\n 'host_network': None,\n 'host_pid': None,\n 'host_users': None,\n 'hostname': None,\n 'image_pull_secrets': [{'name': 'default-dockercfg-6qcsh'}],\n 'init_containers': [{'args': ['hf://facebook/opt-125m',\n '/mnt/models',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-1',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-2'],\n 'command': None,\n 'env': [{'name': 'AWS_ACCESS_KEY_ID',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': None,\n 'secret_key_ref': {'key': 'AWS_ACCESS_KEY_ID',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'AWS_SECRET_ACCESS_KEY',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': N [e2e-llm-inference-service] one,\n 'secret_key_ref': {'key': 'AWS_SECRET_ACCESS_KEY',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'S3_USE_HTTPS',\n 'value': '0',\n 'value_from': None},\n {'name': 'S3_ENDPOINT',\n 'value': 's3-service.kserve:8333',\n 'value_from': None},\n {'name': 'AWS_ENDPOINT_URL',\n 'value': 'http://s3-service.kserve:8333',\n 'value_from': None},\n {'name': 'S3_VERIFY_SSL',\n 'value': '0',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE',\n 'value': '/etc/ssl/custom-certs/cabundle.crt',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE_CONFIGMAP',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'HF_HUB_ENABLE_HF_TRANSFER',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_HIGH_PERFORMANCE',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_NUM_CONCURRENT_RANGE_GETS',\n 'value': '8',\n 'value_from': None},\n {'name': 'CA_BUNDLE_CONFIGMAP_NAME',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'CA_BUNDLE_VOLUME_MOUNT_POINT',\n 'value': '/etc/ssl/custom-certs',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': None,\n 'liveness_probe': None,\n 'name': 'storage-initializer',\n 'ports': None,\n 'readiness_probe': None,\n 'resize_policy': None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n [e2e-llm-inference-service] 'read_only_root_filesystem': None,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': None,\n 'windows_options': None},\n 'startup_probe': None,\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'mount_propagation': None,\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'node_name': 'ip-10-0-131-57.ec2.internal',\n 'node_selector': None,\n 'os': None,\n 'overhead': None,\n 'preemption_policy': 'PreemptLowerPriority',\n 'priority': 0,\n 'priority_class_name': None,\n 'readiness_gates': None,\n 'resource_claims': None,\n 'resources': None,\n 'restart_policy': 'Always',\n 'runtime_class_name': None,\n 'scheduler_name': 'default-scheduler',\n 'scheduling_gates': None,\n 'security_context': {'app_armor_profile': None,\n 'fs_group': 1000690000,\n 'fs_group_change_policy': None,\n 'run_as_group': None,\n 'run_as_non_root': None,\n 'run_as_user': None,\n 'se_linux_change_policy': None,\n 'se_linux_options': {'level': 's0:c26,c20',\n 'role': None,\n 'type': None,\n 'user': None},\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'supplemental_groups': None,\n [e2e-llm-inference-service] 'supplemental_groups_policy': None,\n 'sysctls': None,\n 'windows_options': None},\n 'service_account': 'default',\n 'service_account_name': 'default',\n 'set_hostname_as_fqdn': None,\n 'share_process_namespace': None,\n 'subdomain': None,\n 'termination_grace_period_seconds': 60,\n 'tolerations': [{'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/not-ready',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/unreachable',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoSchedule',\n 'key': 'node.kubernetes.io/memory-pressure',\n 'operator': 'Exists',\n 'toleration_seconds': None,\n 'value': None}],\n 'topology_spread_constraints': None,\n 'volumes': [{'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'home',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': 'Memory', 'size_limit': '1Gi'},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'dshm',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_b [e2e-llm-inference-service] lock_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'model-cache',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tmp-dir',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tls-certs',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': {'default_mode': 420,\n 'items': None,\n [e2e-llm-inference-service] 'optional': None,\n 'secret_name': 'lora-multiple-adapters-test-kserve-self-signed-certs'},\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kserve-provision-location',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': {'default_mode': 420,\n 'items': None,\n 'name': 'odh-kserve-custom-ca-bundle',\n 'optional': None},\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'cabundle-cert',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kube-api-access-n5dws',\n [e2e-llm-inference-service] 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': {'default_mode': 420,\n 'sources': [{'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': {'audience': None,\n 'expiration_seconds': 3607,\n 'path': 'token'}},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'ca.crt',\n 'mode': None,\n 'path': 'ca.crt'}],\n 'name': 'kube-root-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': {'items': [{'field_ref': {'api_version': 'v1',\n 'field_path': 'metadata.namespace'},\n 'mode': None,\n 'path': 'namespace',\n 'resource_field_ref': None}]},\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'service-ca.crt',\n 'mode': None,\n 'path': 'service-ca.crt'}],\n 'name': 'openshift-service-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None}]},\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None}]},\n 'status': {'conditions': [{'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 18, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodReadyToStartContainers'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal()),\n 'message': None,\n [e2e-llm-inference-service] 'reason': None,\n 'status': 'True',\n 'type': 'Initialized'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'Ready'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'ContainersReady'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodScheduled'}],\n 'container_statuses': [{'allocated_resources': {'cpu': '200m',\n 'memory': '2Gi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402',\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_id': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'main',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2',\n 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_count': 0,\n 'started': True,\n 'state': {'running': {'started_at': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal())},\n 'terminated': None,\n 'waiting': None},\n 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/home',\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/tmp',\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/dev/shm',\n 'name': 'dshm',\n 'read_only': None,\n [e2e-llm-inference-service] 'recursive_read_only': None},\n {'mount_path': '/models',\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/var/run/kserve/tls',\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'ephemeral_container_statuses': None,\n 'host_i_ps': [{'ip': '10.0.131.57'}],\n 'host_ip': '10.0.131.57',\n 'init_container_statuses': [{'allocated_resources': {'cpu': '100m',\n 'memory': '100Mi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_id': 'quay.io/opendatahub/kserve-storage-initializer@sha256:60e2a5c8616d82c8933b614ae003dd3412a420a29a0be570b6a6439a3d39ab99',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'storage-initializer',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_count': 0,\n 'started': False,\n 'state': {'running': None,\n 'terminated': {'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'exit_code': 0,\n 'finished_at': datetime.datetime(2026, 7, 6, 21, 35, 22, tzinfo=tzlocal()),\n 'message': None,\n 'reason': 'Completed',\n 'signal': None,\n 'started_at': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n [e2e-llm-inference-service] 'waiting': None},\n 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'message': None,\n 'nominated_node_name': None,\n 'phase': 'Running',\n 'pod_i_ps': [{'ip': '10.134.0.76'}],\n 'pod_ip': '10.134.0.76',\n 'qos_class': 'Burstable',\n 'reason': None,\n 'resize': None,\n 'resource_claim_statuses': None,\n 'start_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())}}] = {'api_version': 'v1',\n 'items': [{'api_version': None,\n 'kind': None,\n 'metadata': {'annotations': {'k8s.ovn.org/pod-networks': '{"default":{"ip_addresses":["10.134.0.76/23"],"mac_address":"0a:58:0a:86:00:4c","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.76/23","gateway_ip":"10.134.0.1","role":"primary"}}',\n 'k8s.v1.cni.cncf.io/network-status': '[{\n'\n ' '\n '"name": '\n '"ovn-kubernetes",\n'\n ' '\n '"interface": '\n '"eth0",\n'\n ' '\n '"ips": '\n '[\n'\n ' '\n '"10.134.0.76"\n'\n ' '\n '],\n'\n ' '\n '"mac": '\n '"0a:58:0a:86:00:4c",\n'\n ' '\n [e2e-llm-inference-service] '"default": '\n 'true,\n'\n ' '\n '"dns": '\n '{}\n'\n '}]',\n 'openshift.io/scc': 'restricted-v2',\n 'seccomp.security.alpha.kubernetes.io/pod': 'runtime/default',\n 'security.openshift.io/validated-scc-subject-type': 'user'},\n 'creation_timestamp': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'deletion_grace_period_seconds': 60,\n 'deletion_timestamp': datetime.datetime(2026, 7, 6, 21, 52, 53, tzinfo=tzlocal()),\n 'finalizers': None,\n 'generate_name': 'lora-multiple-adapters-test-kserve-77485bff95-',\n 'generation': 2,\n 'labels': {'app.kubernetes.io/component': 'llminferenceservice-workload',\n 'app.kubernetes.io/name': 'lora-multiple-adapters-test',\n 'app.kubernetes.io/part-of': 'llminferenceservice',\n 'kserve.io/component': 'workload',\n 'llm-d.ai/role': 'both',\n 'pod-template-hash': '77485bff95'},\n 'managed_fields': [{'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.ovn.org/pod-networks': {}}}},\n 'manager': 'ip-10-0-131-57',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:generateName': {},\n 'f:labels': {'.': {},\n 'f:app.kubernetes.io/component': {},\n 'f:app.kubernetes.io/name': {},\n 'f:app.kubernetes.io/part-of': {},\n 'f:kserve.io/component': {},\n 'f:llm-d.ai/role': {},\n 'f:pod-template-hash': {}},\n 'f:ownerReferences': {'.': {},\n 'k:{"uid":"75c381c4-1742-4fb9-b05f-92ea6db35c9c"}': {}}},\n 'f:spec': {'f:containers': {'k:{"name":"main"}': {'.': {},\n 'f:args': {},\n 'f:command': {},\n [e2e-llm-inference-service] 'f:env': {'.': {},\n 'k:{"name":"HF_HUB_CACHE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HOME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"TORCHINDUCTOR_CACHE_DIR"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"USER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_CPU_KVCACHE_SPACE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_LOGGING_LEVEL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:lifecycle': {'.': {},\n 'f:pr [e2e-llm-inference-service] eStop': {'.': {},\n 'f:exec': {'.': {},\n 'f:command': {}}}},\n 'f:livenessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:name': {},\n 'f:ports': {'.': {},\n 'k:{"containerPort":8000,"protocol":"TCP"}': {'.': {},\n 'f:containerPort': {},\n 'f:protocol': {}}},\n 'f:readinessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:resources': {'.': {},\n [e2e-llm-inference-service] 'f:limits': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:securityContext': {'.': {},\n 'f:allowPrivilegeEscalation': {},\n 'f:capabilities': {'.': {},\n 'f:drop': {}},\n 'f:readOnlyRootFilesystem': {},\n 'f:runAsNonRoot': {},\n 'f:seccompProfile': {'.': {},\n 'f:type': {}}},\n 'f:startupProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:terminationMessagePath': {},\n 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n [e2e-llm-inference-service] 'k:{"mountPath":"/dev/shm"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/home"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/models"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/tmp"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/var/run/kserve/tls"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}}}}},\n 'f:dnsPolicy': {},\n 'f:enableServiceLinks': {},\n 'f:initContainers': {'.': {},\n 'k:{"name":"storage-initializer"}': {'.': {},\n 'f:args': {},\n [e2e-llm-inference-service] 'f:env': {'.': {},\n 'k:{"name":"AWS_ACCESS_KEY_ID"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"AWS_CA_BUNDLE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_ENDPOINT_URL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_SECRET_ACCESS_KEY"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}': {'.': {},\n [e2e-llm-inference-service] 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_HIGH_PERFORMANCE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_ENDPOINT"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_USE_HTTPS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_VERIFY_SSL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n [e2e-llm-inference-service] 'f:name': {},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:terminationMessagePath': {},\n 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/etc/ssl/custom-certs"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}}}}},\n 'f:restartPolicy': {},\n 'f:schedulerName': {},\n 'f:securityContext': {},\n 'f:terminationGracePeriodSeconds': {},\n 'f:volumes': {'.': {},\n 'k:{"name":"cabundle-cert"}': {'.': {},\n 'f:configMap': {'.': {},\n 'f:defaultMode': {} [e2e-llm-inference-service] ,\n 'f:name': {}},\n 'f:name': {}},\n 'k:{"name":"dshm"}': {'.': {},\n 'f:emptyDir': {'.': {},\n 'f:medium': {},\n 'f:sizeLimit': {}},\n 'f:name': {}},\n 'k:{"name":"home"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"kserve-provision-location"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"model-cache"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"tls-certs"}': {'.': {},\n 'f:name': {},\n 'f:secret': {'.': {},\n 'f:defaultMode': {},\n 'f:secretName': {}}},\n 'k:{"name":"tmp-dir"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}}}}},\n 'manager': 'kube-controller-manager',\n 'operation': 'Update',\n 'subresource': None,\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.v1.cni.cncf.io/network-status': {}}}},\n 'manager': 'multus-daemon',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\ [e2e-llm-inference-service] n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:status': {'f:conditions': {'k:{"type":"ContainersReady"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"Initialized"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodReadyToStartContainers"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodScheduled"}': {'f:observedGeneration': {}},\n 'k:{"type":"Ready"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}}},\n 'f:containerStatuses': {},\n 'f:hostIP': {},\n 'f:hostIPs': {},\n 'f:initContainerStatuses': {},\n [e2e-llm-inference-service] 'f:observedGeneration': {},\n 'f:phase': {},\n 'f:podIP': {},\n 'f:podIPs': {'.': {},\n 'k:{"ip":"10.134.0.76"}': {'.': {},\n 'f:ip': {}}},\n 'f:startTime': {}}},\n 'manager': 'kubelet',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 51, 53, tzinfo=tzlocal())}],\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95-2tx5t',\n 'namespace': 'kserve-ci-e2e-test',\n 'owner_references': [{'api_version': 'apps/v1',\n 'block_owner_deletion': True,\n 'controller': True,\n 'kind': 'ReplicaSet',\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95',\n 'uid': '75c381c4-1742-4fb9-b05f-92ea6db35c9c'}],\n 'resource_version': '79769',\n 'self_link': None,\n 'uid': 'd4948f86-f0d1-4911-908f-7cb88db94468'},\n 'spec': {'active_deadline_seconds': None,\n 'affinity': None,\n 'automount_service_account_token': None,\n 'containers': [{'args': ['--enable-lora',\n '--max-lora-rank=64',\n '--max-loras=2',\n '--max-cpu-loras=4',\n '--lora-modules',\n '\'{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\''],\n 'command': ['/bin/bash',\n '-c',\n 'if [ -f '\n '/etc/profile.d/ibm-aiu-setup.sh '\n ']; then\n'\n ' source '\n '/etc/profile.d/ibm-aiu-setup.sh\n'\n 'fi\n'\n '\n'\n 'if [ "$KSERVE_INFER_ROCE" = '\n '"true" ]; then\n'\n ' echo "Trying to infer RoCE '\n 'configs ... "\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gids/* '\n '2>/dev/null\n'\n ' grep -H . '\n '/sys/class/inf [e2e-llm-inference-service] iniband/*/ports/*/gid_attrs/types/* '\n '2>/dev/null\n'\n '\n'\n ' cat '\n '/proc/driver/nvidia/params\n'\n '\n'\n ' '\n 'KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE '\n 'v2"}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'Discovering active HCAs '\n '..."\n'\n ' active_hcas=()\n'\n ' # Loop through all mlx5 '\n 'devices found in sysfs\n'\n ' for hca_dir in '\n '/sys/class/infiniband/mlx5_*; '\n 'do\n'\n " # Ensure it's a "\n 'directory before proceeding\n'\n ' if [ -d "$hca_dir" ]; '\n 'then\n'\n ' '\n 'hca_name=$(basename '\n '"$hca_dir")\n'\n ' '\n 'port_state_file="$hca_dir/ports/1/state" '\n '# Assume port 1\n'\n ' '\n 'type_file="$hca_dir/ports/1/gid_attrs/types/*"\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'Check if the port state file '\n '${port_state_file} exists '\n 'and contains \'ACTIVE\'"\n'\n ' if [ -f '\n '"$port_state_file" ] && grep '\n '-q "ACTIVE" '\n '"$port_state_file" && grep '\n '-q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '${type_file} 2>/dev/null; '\n 'then\n'\n ' echo "[Infer '\n 'RoCE] Found active HCA: '\n '$hca_name"\n'\n ' '\n 'active_hcas+=("$hca_name")\n'\n ' else\n'\n ' echo "[Infer '\n 'RoCE] Skipping inactive or '\n 'down HCA: $hca_name"\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n '\n'\n [e2e-llm-inference-service] ' # Check if we found any '\n 'active HCAs\n'\n ' if [ ${#active_hcas[@]} '\n '-gt 0 ]; then\n'\n ' # Join the array '\n 'elements with a comma\n'\n ' hca_port_pairs=()\n'\n ' for hca in '\n '"${active_hcas[@]}"; do\n'\n ' '\n 'hca_port_pairs+=("${hca}:1")\n'\n ' done\n'\n '\n'\n ' '\n 'active_hca_list=$(IFS=,; '\n 'echo "${active_hcas[*]}")\n'\n ' '\n 'hca_port_pairs_list=$(IFS=,; '\n 'echo '\n '"${hca_port_pairs[*]}")\n'\n ' echo "[Infer RoCE] '\n 'Setting active HCAs: '\n '${active_hca_list}"\n'\n ' export '\n 'NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n'\n ' export '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n'\n ' export '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'NCCL_IB_HCA=${NCCL_IB_HCA}"\n'\n ' echo "[Infer RoCE] '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n'\n ' echo "[Infer RoCE] '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n'\n ' else\n'\n ' echo "[Infer RoCE] '\n 'WARNING: No active RoCE HCAs '\n 'found. NCCL_IB_HCA will not '\n 'be set."\n'\n ' fi\n'\n '\n'\n ' if [ ${#active_hcas[@]} '\n '-gt 0 ]; then\n'\n ' echo "[Infer RoCE] '\n 'Finding GID_INDEX for each '\n 'active HCA (SR-IOV '\n 'compatible)..."\n'\n '\n'\n ' # For SR-IOV '\n 'environments, find the most '\n 'common IPv4 RoCE v2 GID '\n 'index across all HCAs\n'\n ' declare -A '\n [e2e-llm-inference-service] 'gid_index_count\n'\n ' declare -A '\n 'hca_gid_index\n'\n '\n'\n ' for hca_name in '\n '"${active_hcas[@]}"; do\n'\n ' echo "[Infer RoCE] '\n 'Processing HCA: '\n '${hca_name}"\n'\n '\n'\n ' # Find all RoCE v2 '\n 'IPv4 GIDs for this HCA and '\n 'count by index\n'\n ' for tpath in '\n '/sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; '\n 'do\n'\n ' if grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '"$tpath" 2>/dev/null; then\n'\n ' '\n 'idx=$(basename "$tpath")\n'\n ' '\n 'gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n'\n ' # Check '\n 'for IPv4 GID (contains '\n 'ffff:)\n'\n ' if [ -f '\n '"$gid_file" ] && grep -q '\n '"ffff:" "$gid_file"; then\n'\n ' '\n 'gid_value=$(cat "$gid_file" '\n '2>/dev/null || echo "")\n'\n ' echo '\n '"[Infer RoCE] Found IPv4 '\n 'RoCE v2 GID for ${hca_name}: '\n 'index=${idx}, '\n 'gid=${gid_value}"\n'\n ' '\n 'hca_gid_index["${hca_name}"]="${idx}"\n'\n ' '\n 'gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} '\n '+ 1))\n'\n ' break '\n '# Use first found IPv4 GID '\n 'per HCA\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n ' done\n'\n '\n'\n ' # Find the most common '\n 'GID index (most likely to be '\n 'consistent across nodes)\n'\n ' best_gid_index=""\n'\n [e2e-llm-inference-service] ' max_count=0\n'\n ' for idx in '\n '"${!gid_index_count[@]}"; '\n 'do\n'\n ' '\n 'count=${gid_index_count["${idx}"]}\n'\n ' echo "[Infer RoCE] '\n 'GID_INDEX ${idx} found on '\n '${count} HCAs"\n'\n ' if [ $count -gt '\n '$max_count ]; then\n'\n ' '\n 'max_count=$count\n'\n ' '\n 'best_gid_index="$idx"\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Use deterministic '\n 'fallback if tied - prefer '\n 'index 3 (SR-IOV standard)\n'\n ' if [ '\n '${#gid_index_count[@]} -gt 1 '\n ']; then\n'\n ' echo "[Infer RoCE] '\n 'Multiple GID indices found, '\n 'selecting most common: '\n '${best_gid_index}"\n'\n " # If there's a "\n "tie, prefer index 3 as it's "\n 'most common in SR-IOV '\n 'setups\n'\n ' if [ -n '\n '"${gid_index_count[\'3\']}" '\n '] && [ '\n '"${gid_index_count[\'3\']}" '\n '-eq "$max_count" ]; then\n'\n ' '\n 'best_gid_index="3"\n'\n ' echo "[Infer '\n 'RoCE] Using deterministic '\n 'fallback: GID_INDEX=3 '\n '(SR-IOV standard)"\n'\n ' fi\n'\n ' fi\n'\n '\n'\n ' # Check if GID_INDEX '\n 'is already set via '\n 'environment variables\n'\n ' if [ -n '\n '"${NCCL_IB_GID_INDEX}" ]; '\n 'then\n'\n ' echo "[Infer RoCE] '\n 'Using pre-configured '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} '\n 'from environment"\n'\n [e2e-llm-inference-service] ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' echo "[Infer RoCE] '\n 'Using pre-configured '\n 'GID_INDEX=${NCCL_IB_GID_INDEX} '\n 'for NCCL, NVSHMEM, and UCX"\n'\n ' elif [ -n '\n '"$best_gid_index" ]; then\n'\n ' echo "[Infer RoCE] '\n 'Selected GID_INDEX: '\n '${best_gid_index} (found on '\n '${max_count} HCAs)"\n'\n '\n'\n ' export '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'Exported '\n 'GID_INDEX=${best_gid_index} '\n 'for NCCL, NVSHMEM, and UCX"\n'\n ' else\n'\n ' echo "[Infer RoCE] '\n 'ERROR: No valid IPv4 '\n '${KSERVE_INFER_IB_GID_INDEX_GREP} '\n 'GID_INDEX found on any '\n 'HCA."\n'\n ' fi\n'\n ' else\n'\n ' echo "[Infer RoCE] No '\n 'active HCAs found, skipping '\n 'GID_INDEX inference."\n'\n ' fi\n'\n 'fi\n'\n '\n'\n '# '\n '--disable-access-log-for-endpoints '\n 'landed in vLLM 0.16.0 '\n '(vllm-project/vllm#30011).\n'\n '# Older versions still need '\n 'the blanket '\n '--disable-uvicorn-access-log.\n'\n 'ACCESS_LOG_ARGS="--disable-uvicorn-access-log"\n'\n 'VLLM_VERSION=$(vllm '\n '--version 2>/dev/null | tail '\n "-1 | awk '{print $NF}')\n"\n 'echo "[access-log-detect] '\n 'vllm '\n [e2e-llm-inference-service] 'version=\'${VLLM_VERSION}\'"\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ '\n '"$(printf \'%s\\n%s\\n\' '\n '"0.16.0" "${VLLM_VERSION}" | '\n 'sort -V | head -1)" = '\n '"0.16.0" ]; then\n'\n ' '\n 'ACCESS_LOG_ARGS="--disable-access-log-for-endpoints '\n '/health,/metrics,/ping"\n'\n 'fi\n'\n 'echo "[access-log-detect] '\n 'selected '\n 'ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n'\n '\n'\n '# --shutdown-timeout landed '\n 'in vLLM 0.18.0 '\n '(vllm-project/vllm#36666).\n'\n 'SHUTDOWN_TIMEOUT_ARGS=""\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ '\n '"$(printf \'%s\\n%s\\n\' '\n '"0.18.0" "${VLLM_VERSION}" | '\n 'sort -V | head -1)" = '\n '"0.18.0" ]; then\n'\n ' '\n 'SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout '\n '40"\n'\n 'fi\n'\n '\n'\n 'eval "exec vllm serve '\n '/mnt/models \\\n'\n ' --served-model-name '\n '"facebook/opt-125m" '\n '"publishers/kserve-ci-e2e-test/models/facebook/opt-125m" '\n '\\\n'\n ' --port 8000 \\\n'\n ' ${ACCESS_LOG_ARGS} \\\n'\n ' ${SHUTDOWN_TIMEOUT_ARGS} '\n '\\\n'\n ' --enable-ssl-refresh \\\n'\n ' --ssl-certfile '\n '/var/run/kserve/tls/tls.crt '\n '\\\n'\n ' --ssl-keyfile '\n '/var/run/kserve/tls/tls.key '\n '\\\n'\n ' ${VLLM_ADDITIONAL_ARGS} '\n '\\\n'\n ' $@"',\n '--'],\n 'env': [{'name': 'HOME',\n 'value': '/home',\n 'value_from': None},\n {'name': 'VLLM_LOGGING_LEVEL',\n 'value': 'DEBUG',\n 'value_from': None},\n [e2e-llm-inference-service] {'name': 'VLLM_CPU_KVCACHE_SPACE',\n 'value': '1',\n 'value_from': None},\n {'name': 'VLLM_ENABLE_V1_MULTIPROCESSING',\n 'value': '0',\n 'value_from': None},\n {'name': 'USER',\n 'value': 'nonroot',\n 'value_from': None},\n {'name': 'TORCHINDUCTOR_CACHE_DIR',\n 'value': '/tmp/torchinductor-cache',\n 'value_from': None},\n {'name': 'HF_HUB_CACHE',\n 'value': '/models',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': {'post_start': None,\n 'pre_stop': {'_exec': {'command': ['/bin/sleep',\n '15']},\n 'http_get': None,\n 'sleep': None,\n 'tcp_socket': None}},\n 'liveness_probe': {'_exec': None,\n 'failure_threshold': 10,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'name': 'main',\n 'ports': [{'container_port': 8000,\n 'host_ip': None,\n 'host_port': None,\n 'name': None,\n 'protocol': 'TCP'}],\n 'readiness_probe': {'_exec': None,\n 'failure_threshold': 2,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n [e2e-llm-inference-service] 'initial_delay_seconds': None,\n 'period_seconds': 1,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'resize_policy': None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2',\n 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n 'read_only_root_filesystem': True,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'windows_options': None},\n 'startup_probe': {'_exec': None,\n 'failure_threshold': 60,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/home',\n 'mount_propagation': None,\n 'name': 'home',\n 'read_only': None,\n [e2e-llm-inference-service] 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/tmp',\n 'mount_propagation': None,\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/dev/shm',\n 'mount_propagation': None,\n 'name': 'dshm',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/models',\n 'mount_propagation': None,\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/kserve/tls',\n 'mount_propagation': None,\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'dns_config': None,\n 'dns_policy': 'ClusterFirst',\n 'enable_service_links': True,\n 'ephemeral_containers': None,\n 'host_aliases': None,\n 'host_ipc': None,\n 'host_network': None,\n 'host_pid' [e2e-llm-inference-service] : None,\n 'host_users': None,\n 'hostname': None,\n 'image_pull_secrets': [{'name': 'default-dockercfg-6qcsh'}],\n 'init_containers': [{'args': ['hf://facebook/opt-125m',\n '/mnt/models',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-1',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-2'],\n 'command': None,\n 'env': [{'name': 'AWS_ACCESS_KEY_ID',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': None,\n 'secret_key_ref': {'key': 'AWS_ACCESS_KEY_ID',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'AWS_SECRET_ACCESS_KEY',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': None,\n 'secret_key_ref': {'key': 'AWS_SECRET_ACCESS_KEY',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'S3_USE_HTTPS',\n 'value': '0',\n 'value_from': None},\n {'name': 'S3_ENDPOINT',\n 'value': 's3-service.kserve:8333',\n 'value_from': None},\n {'name': 'AWS_ENDPOINT_URL',\n 'value': 'http://s3-service.kserve:8333',\n 'value_from': None},\n {'name': 'S3_VERIFY_SSL',\n 'value': '0',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE',\n 'value': '/etc/ssl/custom-certs/cabundle.crt',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE_CONFIGMAP',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'HF_HUB_ENABLE_HF_TRANSFER',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_HIGH_PERFORMANCE',\n 'value': '1',\n 'value_fro [e2e-llm-inference-service] m': None},\n {'name': 'HF_XET_NUM_CONCURRENT_RANGE_GETS',\n 'value': '8',\n 'value_from': None},\n {'name': 'CA_BUNDLE_CONFIGMAP_NAME',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'CA_BUNDLE_VOLUME_MOUNT_POINT',\n 'value': '/etc/ssl/custom-certs',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': None,\n 'liveness_probe': None,\n 'name': 'storage-initializer',\n 'ports': None,\n 'readiness_probe': None,\n 'resize_policy': None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n 'read_only_root_filesystem': None,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': None,\n 'windows_options': None},\n 'startup_probe': None,\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None,\n [e2e-llm-inference-service] 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'mount_propagation': None,\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'node_name': 'ip-10-0-131-57.ec2.internal',\n 'node_selector': None,\n 'os': None,\n 'overhead': None,\n 'preemption_policy': 'PreemptLowerPriority',\n 'priority': 0,\n 'priority_class_name': None,\n 'readiness_gates': None,\n 'resource_claims': None,\n 'resources': None,\n 'restart_policy': 'Always',\n 'runtime_class_name': None,\n 'scheduler_name': 'default-scheduler',\n 'scheduling_gates': None,\n 'security_context': {'app_armor_profile': None,\n 'fs_group': 1000690000,\n 'fs_group_change_policy': None,\n 'run_as_group': None,\n 'run_as_non_root': None,\n 'run_as_user': None,\n 'se_linux_change_policy': None,\n 'se_linux_options': {'level': 's0:c26,c20',\n 'role': None,\n 'type': None,\n 'user': None},\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'supplemental_groups': None,\n 'supplemental_groups_policy': None,\n 'sysctls': None,\n 'windows_options': None},\n 'service_account': 'default',\n 'service_account_name': 'default',\n 'set_hostname_as_fqdn': None,\n 'share_process_namespace': None,\n 'subdomain': None,\n 'termination_grace_period_seconds': 60,\n 'tolerations': [{'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/not-ready',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n [e2e-llm-inference-service] {'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/unreachable',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoSchedule',\n 'key': 'node.kubernetes.io/memory-pressure',\n 'operator': 'Exists',\n 'toleration_seconds': None,\n 'value': None}],\n 'topology_spread_constraints': None,\n 'volumes': [{'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None,\n 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'home',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': 'Memory',\n 'size_limit': '1Gi'},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'dshm',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n [e2e-llm-inference-service] 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None,\n 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'model-cache',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None,\n 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tmp-dir',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n [e2e-llm-inference-service] 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tls-certs',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': {'default_mode': 420,\n 'items': None,\n 'optional': None,\n 'secret_name': 'lora-multiple-adapters-test-kserve-self-signed-certs'},\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None,\n 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kserve-provision-location',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': {'default_mode': 420,\n 'items': None,\n 'name': 'odh-kserve-custom-ca-bundle',\n 'optional': None},\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\ [e2e-llm-inference-service] n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'cabundle-cert',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kube-api-access-n5dws',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': {'default_mode': 420,\n 'sources': [{'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': {'audience': None,\n 'expiration_seconds': 3607,\n 'path': 'token'}},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'ca.crt',\n 'mode': None,\n 'path': 'ca.crt'}],\n 'name': 'kube-root-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n [e2e-llm-inference-service] 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': {'items': [{'field_ref': {'api_version': 'v1',\n 'field_path': 'metadata.namespace'},\n 'mode': None,\n 'path': 'namespace',\n 'resource_field_ref': None}]},\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'service-ca.crt',\n 'mode': None,\n 'path': 'service-ca.crt'}],\n 'name': 'openshift-service-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None}]},\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None}]},\n 'status': {'conditions': [{'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 18, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodReadyToStartContainers'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'Initialized'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'Ready'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'ContainersReady'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzin [e2e-llm-inference-service] fo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodScheduled'}],\n 'container_statuses': [{'allocated_resources': {'cpu': '200m',\n 'memory': '2Gi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402',\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_id': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'main',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2',\n 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_count': 0,\n 'started': True,\n 'state': {'running': {'started_at': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal())},\n 'terminated': None,\n 'waiting': None},\n 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/home',\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/tmp',\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/dev/shm',\n 'name': 'dshm',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/models',\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/var/run/kserve/tls',\n [e2e-llm-inference-service] 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'ephemeral_container_statuses': None,\n 'host_i_ps': [{'ip': '10.0.131.57'}],\n 'host_ip': '10.0.131.57',\n 'init_container_statuses': [{'allocated_resources': {'cpu': '100m',\n 'memory': '100Mi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_id': 'quay.io/opendatahub/kserve-storage-initializer@sha256:60e2a5c8616d82c8933b614ae003dd3412a420a29a0be570b6a6439a3d39ab99',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'storage-initializer',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_count': 0,\n 'started': False,\n 'state': {'running': None,\n 'terminated': {'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'exit_code': 0,\n 'finished_at': datetime.datetime(2026, 7, 6, 21, 35, 22, tzinfo=tzlocal()),\n 'message': None,\n 'reason': 'Completed',\n 'signal': None,\n 'started_at': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n 'waiting': None},\n [e2e-llm-inference-service] 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'message': None,\n 'nominated_node_name': None,\n 'phase': 'Running',\n 'pod_i_ps': [{'ip': '10.134.0.76'}],\n 'pod_ip': '10.134.0.76',\n 'qos_class': 'Burstable',\n 'reason': None,\n 'resize': None,\n 'resource_claim_statuses': None,\n 'start_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())}}],\n 'kind': 'PodList',\n 'metadata': {'_continue': None,\n 'remaining_item_count': None,\n 'resource_version': '79926',\n 'self_link': None}}.items [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: 1 pod(s) for lora-multiple-adapters-test still terminating [e2e-llm-inference-service] assert not [{'api_version': None,\n 'kind': None,\n 'metadata': {'annotations': {'k8s.ovn.org/pod-networks': '{"default":{"ip_addresses":["10.134.0.76/23"],"mac_address":"0a:58:0a:86:00:4c","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.76/23","gateway_ip":"10.134.0.1","role":"primary"}}',\n 'k8s.v1.cni.cncf.io/network-status': '[{\n'\n ' '\n '"name": '\n '"ovn-kubernetes",\n'\n ' '\n '"interface": '\n '"eth0",\n'\n ' "ips": '\n '[\n'\n ' '\n '"10.134.0.76"\n'\n ' ],\n'\n ' "mac": '\n '"0a:58:0a:86:00:4c",\n'\n ' '\n '"default": '\n 'true,\n'\n ' "dns": '\n '{}\n'\n '}]',\n 'openshift.io/scc': 'restricted-v2',\n 'seccomp.security.alpha.kubernetes.io/pod': 'runtime/default',\n 'security.openshift.io/validated-scc-subject-type': 'user'},\n 'creation_timestamp': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'deletion_grace_period_seconds': 60,\n 'deletion_timestamp': datetime.datetime(2026, 7, 6, 21, 52, 53, tzinfo=tzlocal()),\n 'finalizers': None,\n 'generate_name': 'lora-multiple-adapters-test-kserve-77485bff95-',\n 'generation': 2,\n 'labels': {'app.kubernetes.io/component': 'llminferenceservice-workload',\n 'app.kubernetes.io/name': 'lora-multiple-adapters-test',\n 'app.kubernetes.io/part-of': 'llminferenceservice',\n 'kserve.io/component': 'workload',\n 'llm-d.ai/role': 'both',\n 'pod-template-hash': '77485bff95'},\n 'managed_fields': [{'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.ovn.org/pod-networks': {}}}},\n 'manager': 'ip-10-0-131-57',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:generateName': {},\n 'f:labels': {'.': {},\n 'f:app.kubernetes.io/component' [e2e-llm-inference-service] : {},\n 'f:app.kubernetes.io/name': {},\n 'f:app.kubernetes.io/part-of': {},\n 'f:kserve.io/component': {},\n 'f:llm-d.ai/role': {},\n 'f:pod-template-hash': {}},\n 'f:ownerReferences': {'.': {},\n 'k:{"uid":"75c381c4-1742-4fb9-b05f-92ea6db35c9c"}': {}}},\n 'f:spec': {'f:containers': {'k:{"name":"main"}': {'.': {},\n 'f:args': {},\n 'f:command': {},\n 'f:env': {'.': {},\n 'k:{"name":"HF_HUB_CACHE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HOME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"TORCHINDUCTOR_CACHE_DIR"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"USER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_CPU_KVCACHE_SPACE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n [e2e-llm-inference-service] 'k:{"name":"VLLM_LOGGING_LEVEL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:lifecycle': {'.': {},\n 'f:preStop': {'.': {},\n 'f:exec': {'.': {},\n 'f:command': {}}}},\n 'f:livenessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:name': {},\n 'f:ports': {'.': {},\n 'k:{"containerPort":8000,"protocol":"TCP"}': {'.': {},\n 'f:containerPort': {},\n 'f:protocol': {}}},\n 'f:readinessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n [e2e-llm-inference-service] 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:securityContext': {'.': {},\n 'f:allowPrivilegeEscalation': {},\n 'f:capabilities': {'.': {},\n 'f:drop': {}},\n 'f:readOnlyRootFilesystem': {},\n 'f:runAsNonRoot': {},\n 'f:seccompProfile': {'.': {},\n 'f:type': {}}},\n 'f:startupProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:terminationMessagePath': {},\n [e2e-llm-inference-service] 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/dev/shm"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/home"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/models"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/tmp"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/var/run/kserve/tls"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}}}}},\n 'f:dnsPolicy': {},\n 'f:enableServiceLinks': {},\n 'f:initContainers': {'.': {},\n 'k:{"name":"storage-initializer"}': {'.': {},\n 'f:args': {},\n [e2e-llm-inference-service] 'f:env': {'.': {},\n 'k:{"name":"AWS_ACCESS_KEY_ID"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"AWS_CA_BUNDLE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_ENDPOINT_URL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_SECRET_ACCESS_KEY"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}': {'.': {},\n 'f:name': {},\n [e2e-llm-inference-service] 'f:value': {}},\n 'k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_HIGH_PERFORMANCE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_ENDPOINT"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_USE_HTTPS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_VERIFY_SSL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:name': {},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n [e2e-llm-inference-service] 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:terminationMessagePath': {},\n 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/etc/ssl/custom-certs"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}}}}},\n 'f:restartPolicy': {},\n 'f:schedulerName': {},\n 'f:securityContext': {},\n 'f:terminationGracePeriodSeconds': {},\n 'f:volumes': {'.': {},\n 'k:{"name":"cabundle-cert"}': {'.': {},\n 'f:configMap': {'.': {},\n 'f:defaultMode': {},\n 'f:name': {}},\n 'f:name': {}},\n 'k:{"name":"dshm"}': {'.': {},\n 'f:emptyDir': {'.': {},\n 'f:medium': {},\n [e2e-llm-inference-service] 'f:sizeLimit': {}},\n 'f:name': {}},\n 'k:{"name":"home"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"kserve-provision-location"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"model-cache"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"tls-certs"}': {'.': {},\n 'f:name': {},\n 'f:secret': {'.': {},\n 'f:defaultMode': {},\n 'f:secretName': {}}},\n 'k:{"name":"tmp-dir"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}}}}},\n 'manager': 'kube-controller-manager',\n 'operation': 'Update',\n 'subresource': None,\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.v1.cni.cncf.io/network-status': {}}}},\n 'manager': 'multus-daemon',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:status': {'f:conditions': {'k:{"type":"ContainersReady"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"Initialized"}': {'.': {},\n [e2e-llm-inference-service] 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodReadyToStartContainers"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodScheduled"}': {'f:observedGeneration': {}},\n 'k:{"type":"Ready"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}}},\n 'f:containerStatuses': {},\n 'f:hostIP': {},\n 'f:hostIPs': {},\n 'f:initContainerStatuses': {},\n 'f:observedGeneration': {},\n 'f:phase': {},\n 'f:podIP': {},\n 'f:podIPs': {'.': {},\n 'k:{"ip":"10.134.0.76"}': {'.': {},\n 'f:ip': {}}},\n 'f:startTime': {}}},\n 'manager': 'kubelet',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 51, 53, tzinfo=tzlocal())}],\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95-2tx5t',\n 'namespace': 'kserve-ci-e2e-test',\n 'owner_references': [{'api_version': 'apps/v1',\n 'block_owner_deletion': True,\n 'controller': True,\n 'kind': 'ReplicaSet',\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95',\n 'uid': '75c381c4-1742-4fb9-b05f-92ea6db3 [e2e-llm-inference-service] 5c9c'}],\n 'resource_version': '79769',\n 'self_link': None,\n 'uid': 'd4948f86-f0d1-4911-908f-7cb88db94468'},\n 'spec': {'active_deadline_seconds': None,\n 'affinity': None,\n 'automount_service_account_token': None,\n 'containers': [{'args': ['--enable-lora',\n '--max-lora-rank=64',\n '--max-loras=2',\n '--max-cpu-loras=4',\n '--lora-modules',\n '\'{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\''],\n 'command': ['/bin/bash',\n '-c',\n 'if [ -f /etc/profile.d/ibm-aiu-setup.sh '\n ']; then\n'\n ' source '\n '/etc/profile.d/ibm-aiu-setup.sh\n'\n 'fi\n'\n '\n'\n 'if [ "$KSERVE_INFER_ROCE" = "true" ]; '\n 'then\n'\n ' echo "Trying to infer RoCE configs '\n '... "\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gids/* '\n '2>/dev/null\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gid_attrs/types/* '\n '2>/dev/null\n'\n '\n'\n ' cat /proc/driver/nvidia/params\n'\n '\n'\n ' '\n 'KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE '\n 'v2"}\n'\n '\n'\n ' echo "[Infer RoCE] Discovering active '\n 'HCAs ..."\n'\n ' active_hcas=()\n'\n ' # Loop through all mlx5 devices found '\n 'in sysfs\n'\n ' for hca_dir in '\n '/sys/class/infiniband/mlx5_*; do\n'\n " # Ensure it's a directory before "\n 'proceeding\n'\n ' if [ -d "$hca_dir" ]; then\n'\n ' hca_name=$(basename '\n '"$hca_dir")\n'\n ' '\n 'port_state_file="$hca_dir/ports/1/state" '\n '# Assume port 1\n'\n ' '\n 'type_file="$hca_dir/ports/1/gid_attrs/types/*"\n'\n '\n'\n ' echo "[Infer RoCE] Check if '\n 'the port state file ${port_state_file} '\n 'exists and contains \'ACTIVE\'"\n'\n ' if [ -f "$port_state_file" ] '\n '&& grep -q "AC [e2e-llm-inference-service] TIVE" "$port_state_file" '\n '&& grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '${type_file} 2>/dev/null; then\n'\n ' echo "[Infer RoCE] Found '\n 'active HCA: $hca_name"\n'\n ' '\n 'active_hcas+=("$hca_name")\n'\n ' else\n'\n ' echo "[Infer RoCE] '\n 'Skipping inactive or down HCA: '\n '$hca_name"\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Check if we found any active HCAs\n'\n ' if [ ${#active_hcas[@]} -gt 0 ]; '\n 'then\n'\n ' # Join the array elements with a '\n 'comma\n'\n ' hca_port_pairs=()\n'\n ' for hca in "${active_hcas[@]}"; '\n 'do\n'\n ' hca_port_pairs+=("${hca}:1")\n'\n ' done\n'\n '\n'\n ' active_hca_list=$(IFS=,; echo '\n '"${active_hcas[*]}")\n'\n ' hca_port_pairs_list=$(IFS=,; echo '\n '"${hca_port_pairs[*]}")\n'\n ' echo "[Infer RoCE] Setting active '\n 'HCAs: ${active_hca_list}"\n'\n ' export '\n 'NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n'\n ' export '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n'\n ' export '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'NCCL_IB_HCA=${NCCL_IB_HCA}"\n'\n ' echo "[Infer RoCE] '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n'\n ' echo "[Infer RoCE] '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n'\n ' else\n'\n ' echo "[Infer RoCE] WARNING: No '\n 'active RoCE HCAs found. NCCL_IB_HCA '\n 'will not be set."\n'\n ' fi\n'\n '\n'\n ' if [ ${#active_hcas[@]} -gt 0 ]; '\n 'then\n'\n ' echo "[Infer RoCE] Finding '\n 'GID_INDEX for each active HCA (SR-IOV '\n 'compatible)..."\n'\n '\n'\n ' # For SR-IOV environments, find '\n 'the most common IPv4 RoCE v2 GID index '\n 'across all HCAs\n'\n ' declare -A gid_index_count\n'\n [e2e-llm-inference-service] ' declare -A hca_gid_index\n'\n '\n'\n ' for hca_name in '\n '"${active_hcas[@]}"; do\n'\n ' echo "[Infer RoCE] Processing '\n 'HCA: ${hca_name}"\n'\n '\n'\n ' # Find all RoCE v2 IPv4 GIDs '\n 'for this HCA and count by index\n'\n ' for tpath in '\n '/sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; '\n 'do\n'\n ' if grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '"$tpath" 2>/dev/null; then\n'\n ' idx=$(basename '\n '"$tpath")\n'\n ' '\n 'gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n'\n ' # Check for IPv4 GID '\n '(contains ffff:)\n'\n ' if [ -f "$gid_file" ] '\n '&& grep -q "ffff:" "$gid_file"; then\n'\n ' gid_value=$(cat '\n '"$gid_file" 2>/dev/null || echo "")\n'\n ' echo "[Infer '\n 'RoCE] Found IPv4 RoCE v2 GID for '\n '${hca_name}: index=${idx}, '\n 'gid=${gid_value}"\n'\n ' '\n 'hca_gid_index["${hca_name}"]="${idx}"\n'\n ' '\n 'gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} '\n '+ 1))\n'\n ' break # Use '\n 'first found IPv4 GID per HCA\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n ' done\n'\n '\n'\n ' # Find the most common GID index '\n '(most likely to be consistent across '\n 'nodes)\n'\n ' best_gid_index=""\n'\n ' max_count=0\n'\n ' for idx in '\n '"${!gid_index_count[@]}"; do\n'\n ' '\n 'count=${gid_index_count["${idx}"]}\n'\n ' echo "[Infer RoCE] GID_INDEX '\n '${idx} found on ${count} HCAs"\n'\n ' if [ $count -gt $max_count ]; '\n 'then\n'\n ' max_count=$count\n'\n ' best_gid_index="$idx"\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Use determini [e2e-llm-inference-service] stic fallback if '\n 'tied - prefer index 3 (SR-IOV '\n 'standard)\n'\n ' if [ ${#gid_index_count[@]} -gt 1 '\n ']; then\n'\n ' echo "[Infer RoCE] Multiple '\n 'GID indices found, selecting most '\n 'common: ${best_gid_index}"\n'\n " # If there's a tie, prefer "\n "index 3 as it's most common in SR-IOV "\n 'setups\n'\n ' if [ -n '\n '"${gid_index_count[\'3\']}" ] && [ '\n '"${gid_index_count[\'3\']}" -eq '\n '"$max_count" ]; then\n'\n ' best_gid_index="3"\n'\n ' echo "[Infer RoCE] Using '\n 'deterministic fallback: GID_INDEX=3 '\n '(SR-IOV standard)"\n'\n ' fi\n'\n ' fi\n'\n '\n'\n ' # Check if GID_INDEX is already '\n 'set via environment variables\n'\n ' if [ -n "${NCCL_IB_GID_INDEX}" ]; '\n 'then\n'\n ' echo "[Infer RoCE] Using '\n 'pre-configured '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} '\n 'from environment"\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' echo "[Infer RoCE] Using '\n 'pre-configured '\n 'GID_INDEX=${NCCL_IB_GID_INDEX} for '\n 'NCCL, NVSHMEM, and UCX"\n'\n ' elif [ -n "$best_gid_index" ]; '\n 'then\n'\n ' echo "[Infer RoCE] Selected '\n 'GID_INDEX: ${best_gid_index} (found on '\n '${max_count} HCAs)"\n'\n '\n'\n ' export '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n'\n '\n'\n ' echo "[Infer RoCE] Exported '\n 'GID_INDEX=${best_gid_index} for NCCL, '\n 'NVSHMEM, and UCX"\n'\n ' else\n'\n ' echo "[Infer RoCE] ERROR: No '\n 'valid IPv4 '\n '${KSERVE_INFER_IB_GID_INDEX_GREP} '\n 'GID_INDEX found on any HCA."\n'\n [e2e-llm-inference-service] ' fi\n'\n ' else\n'\n ' echo "[Infer RoCE] No active HCAs '\n 'found, skipping GID_INDEX inference."\n'\n ' fi\n'\n 'fi\n'\n '\n'\n '# --disable-access-log-for-endpoints '\n 'landed in vLLM 0.16.0 '\n '(vllm-project/vllm#30011).\n'\n '# Older versions still need the blanket '\n '--disable-uvicorn-access-log.\n'\n 'ACCESS_LOG_ARGS="--disable-uvicorn-access-log"\n'\n 'VLLM_VERSION=$(vllm --version '\n "2>/dev/null | tail -1 | awk '{print "\n "$NF}')\n"\n 'echo "[access-log-detect] vllm '\n 'version=\'${VLLM_VERSION}\'"\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ "$(printf '\n '\'%s\\n%s\\n\' "0.16.0" '\n '"${VLLM_VERSION}" | sort -V | head -1)" '\n '= "0.16.0" ]; then\n'\n ' '\n 'ACCESS_LOG_ARGS="--disable-access-log-for-endpoints '\n '/health,/metrics,/ping"\n'\n 'fi\n'\n 'echo "[access-log-detect] selected '\n 'ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n'\n '\n'\n '# --shutdown-timeout landed in vLLM '\n '0.18.0 (vllm-project/vllm#36666).\n'\n 'SHUTDOWN_TIMEOUT_ARGS=""\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ "$(printf '\n '\'%s\\n%s\\n\' "0.18.0" '\n '"${VLLM_VERSION}" | sort -V | head -1)" '\n '= "0.18.0" ]; then\n'\n ' '\n 'SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout '\n '40"\n'\n 'fi\n'\n '\n'\n 'eval "exec vllm serve /mnt/models \\\n'\n ' --served-model-name '\n '"facebook/opt-125m" '\n '"publishers/kserve-ci-e2e-test/models/facebook/opt-125m" '\n '\\\n'\n ' --port 8000 \\\n'\n ' ${ACCESS_LOG_ARGS} \\\n'\n ' ${SHUTDOWN_TIMEOUT_ARGS} \\\n'\n ' --enable-ssl-refresh \\\n'\n ' --ssl-certfile '\n '/var/run/kserve/tls/tls.crt \\\n'\n ' --ssl-keyfile '\n '/var/run/kserve/tls/tls.key \\\n'\n ' ${VLLM_ADDITIONAL_ARGS} \\\n'\n ' $@"',\n '--'],\n 'env': [{'name': 'HOME',\n 'value': '/home',\n 'value_from': None},\n {'name': 'VLL [e2e-llm-inference-service] M_LOGGING_LEVEL',\n 'value': 'DEBUG',\n 'value_from': None},\n {'name': 'VLLM_CPU_KVCACHE_SPACE',\n 'value': '1',\n 'value_from': None},\n {'name': 'VLLM_ENABLE_V1_MULTIPROCESSING',\n 'value': '0',\n 'value_from': None},\n {'name': 'USER',\n 'value': 'nonroot',\n 'value_from': None},\n {'name': 'TORCHINDUCTOR_CACHE_DIR',\n 'value': '/tmp/torchinductor-cache',\n 'value_from': None},\n {'name': 'HF_HUB_CACHE',\n 'value': '/models',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': {'post_start': None,\n 'pre_stop': {'_exec': {'command': ['/bin/sleep',\n '15']},\n 'http_get': None,\n 'sleep': None,\n 'tcp_socket': None}},\n 'liveness_probe': {'_exec': None,\n 'failure_threshold': 10,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'name': 'main',\n 'ports': [{'container_port': 8000,\n 'host_ip': None,\n 'host_port': None,\n 'name': None,\n 'protocol': 'TCP'}],\n 'readiness_probe': {'_exec': None,\n 'failure_threshold': 2,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 1,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'resize_policy': [e2e-llm-inference-service] None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2', 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n 'read_only_root_filesystem': True,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'windows_options': None},\n 'startup_probe': {'_exec': None,\n 'failure_threshold': 60,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/home',\n 'mount_propagation': None,\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/tmp',\n 'mount_propagation': None,\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/dev/shm',\n 'mount_propagation': None,\n 'name': 'dshm',\n 'read_only': None,\n 'recursive_read_only': None,\n [e2e-llm-inference-service] 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/models',\n 'mount_propagation': None,\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/kserve/tls',\n 'mount_propagation': None,\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'dns_config': None,\n 'dns_policy': 'ClusterFirst',\n 'enable_service_links': True,\n 'ephemeral_containers': None,\n 'host_aliases': None,\n 'host_ipc': None,\n 'host_network': None,\n 'host_pid': None,\n 'host_users': None,\n 'hostname': None,\n 'image_pull_secrets': [{'name': 'default-dockercfg-6qcsh'}],\n 'init_containers': [{'args': ['hf://facebook/opt-125m',\n '/mnt/models',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-1',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-2'],\n 'command': None,\n 'env': [{'name': 'AWS_ACCESS_KEY_ID',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': None,\n 'secret_key_ref': {'key': 'AWS_ACCESS_KEY_ID',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'AWS_SECRET_ACCESS_KEY',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': [e2e-llm-inference-service] None,\n 'secret_key_ref': {'key': 'AWS_SECRET_ACCESS_KEY',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'S3_USE_HTTPS',\n 'value': '0',\n 'value_from': None},\n {'name': 'S3_ENDPOINT',\n 'value': 's3-service.kserve:8333',\n 'value_from': None},\n {'name': 'AWS_ENDPOINT_URL',\n 'value': 'http://s3-service.kserve:8333',\n 'value_from': None},\n {'name': 'S3_VERIFY_SSL',\n 'value': '0',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE',\n 'value': '/etc/ssl/custom-certs/cabundle.crt',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE_CONFIGMAP',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'HF_HUB_ENABLE_HF_TRANSFER',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_HIGH_PERFORMANCE',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_NUM_CONCURRENT_RANGE_GETS',\n 'value': '8',\n 'value_from': None},\n {'name': 'CA_BUNDLE_CONFIGMAP_NAME',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'CA_BUNDLE_VOLUME_MOUNT_POINT',\n 'value': '/etc/ssl/custom-certs',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': None,\n 'liveness_probe': None,\n 'name': 'storage-initializer',\n 'ports': None,\n 'readiness_probe': None,\n 'resize_policy': None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n [e2e-llm-inference-service] 'read_only_root_filesystem': None,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': None,\n 'windows_options': None},\n 'startup_probe': None,\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'mount_propagation': None,\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'node_name': 'ip-10-0-131-57.ec2.internal',\n 'node_selector': None,\n 'os': None,\n 'overhead': None,\n 'preemption_policy': 'PreemptLowerPriority',\n 'priority': 0,\n 'priority_class_name': None,\n 'readiness_gates': None,\n 'resource_claims': None,\n 'resources': None,\n 'restart_policy': 'Always',\n 'runtime_class_name': None,\n 'scheduler_name': 'default-scheduler',\n 'scheduling_gates': None,\n 'security_context': {'app_armor_profile': None,\n 'fs_group': 1000690000,\n 'fs_group_change_policy': None,\n 'run_as_group': None,\n 'run_as_non_root': None,\n 'run_as_user': None,\n 'se_linux_change_policy': None,\n 'se_linux_options': {'level': 's0:c26,c20',\n 'role': None,\n 'type': None,\n 'user': None},\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'supplemental_groups': None,\n [e2e-llm-inference-service] 'supplemental_groups_policy': None,\n 'sysctls': None,\n 'windows_options': None},\n 'service_account': 'default',\n 'service_account_name': 'default',\n 'set_hostname_as_fqdn': None,\n 'share_process_namespace': None,\n 'subdomain': None,\n 'termination_grace_period_seconds': 60,\n 'tolerations': [{'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/not-ready',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/unreachable',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoSchedule',\n 'key': 'node.kubernetes.io/memory-pressure',\n 'operator': 'Exists',\n 'toleration_seconds': None,\n 'value': None}],\n 'topology_spread_constraints': None,\n 'volumes': [{'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'home',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': 'Memory', 'size_limit': '1Gi'},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'dshm',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_ [e2e-llm-inference-service] block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'model-cache',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tmp-dir',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tls-certs',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': {'default_mode': 420,\n 'items': None,\n [e2e-llm-inference-service] 'optional': None,\n 'secret_name': 'lora-multiple-adapters-test-kserve-self-signed-certs'},\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kserve-provision-location',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': {'default_mode': 420,\n 'items': None,\n 'name': 'odh-kserve-custom-ca-bundle',\n 'optional': None},\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'cabundle-cert',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kube-api-access-n5dws',\n [e2e-llm-inference-service] 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': {'default_mode': 420,\n 'sources': [{'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': {'audience': None,\n 'expiration_seconds': 3607,\n 'path': 'token'}},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'ca.crt',\n 'mode': None,\n 'path': 'ca.crt'}],\n 'name': 'kube-root-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': {'items': [{'field_ref': {'api_version': 'v1',\n 'field_path': 'metadata.namespace'},\n 'mode': None,\n 'path': 'namespace',\n 'resource_field_ref': None}]},\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'service-ca.crt',\n 'mode': None,\n 'path': 'service-ca.crt'}],\n 'name': 'openshift-service-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None}]},\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None}]},\n 'status': {'conditions': [{'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 18, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodReadyToStartContainers'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal()),\n 'message': None,\n [e2e-llm-inference-service] 'reason': None,\n 'status': 'True',\n 'type': 'Initialized'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'Ready'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'ContainersReady'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodScheduled'}],\n 'container_statuses': [{'allocated_resources': {'cpu': '200m',\n 'memory': '2Gi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402',\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_id': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'main',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2',\n 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_count': 0,\n 'started': True,\n 'state': {'running': {'started_at': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal())},\n 'terminated': None,\n 'waiting': None},\n 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/home',\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/tmp',\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/dev/shm',\n 'name': 'dshm',\n 'read_only': None,\n [e2e-llm-inference-service] 'recursive_read_only': None},\n {'mount_path': '/models',\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/var/run/kserve/tls',\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'ephemeral_container_statuses': None,\n 'host_i_ps': [{'ip': '10.0.131.57'}],\n 'host_ip': '10.0.131.57',\n 'init_container_statuses': [{'allocated_resources': {'cpu': '100m',\n 'memory': '100Mi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_id': 'quay.io/opendatahub/kserve-storage-initializer@sha256:60e2a5c8616d82c8933b614ae003dd3412a420a29a0be570b6a6439a3d39ab99',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'storage-initializer',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_count': 0,\n 'started': False,\n 'state': {'running': None,\n 'terminated': {'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'exit_code': 0,\n 'finished_at': datetime.datetime(2026, 7, 6, 21, 35, 22, tzinfo=tzlocal()),\n 'message': None,\n 'reason': 'Completed',\n 'signal': None,\n 'started_at': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n [e2e-llm-inference-service] 'waiting': None},\n 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'message': None,\n 'nominated_node_name': None,\n 'phase': 'Running',\n 'pod_i_ps': [{'ip': '10.134.0.76'}],\n 'pod_ip': '10.134.0.76',\n 'qos_class': 'Burstable',\n 'reason': None,\n 'resize': None,\n 'resource_claim_statuses': None,\n 'start_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())}}] [e2e-llm-inference-service] + where [{'api_version': None,\n 'kind': None,\n 'metadata': {'annotations': {'k8s.ovn.org/pod-networks': '{"default":{"ip_addresses":["10.134.0.76/23"],"mac_address":"0a:58:0a:86:00:4c","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.76/23","gateway_ip":"10.134.0.1","role":"primary"}}',\n 'k8s.v1.cni.cncf.io/network-status': '[{\n'\n ' '\n '"name": '\n '"ovn-kubernetes",\n'\n ' '\n '"interface": '\n '"eth0",\n'\n ' "ips": '\n '[\n'\n ' '\n '"10.134.0.76"\n'\n ' ],\n'\n ' "mac": '\n '"0a:58:0a:86:00:4c",\n'\n ' '\n '"default": '\n 'true,\n'\n ' "dns": '\n '{}\n'\n '}]',\n 'openshift.io/scc': 'restricted-v2',\n 'seccomp.security.alpha.kubernetes.io/pod': 'runtime/default',\n 'security.openshift.io/validated-scc-subject-type': 'user'},\n 'creation_timestamp': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'deletion_grace_period_seconds': 60,\n 'deletion_timestamp': datetime.datetime(2026, 7, 6, 21, 52, 53, tzinfo=tzlocal()),\n 'finalizers': None,\n 'generate_name': 'lora-multiple-adapters-test-kserve-77485bff95-',\n 'generation': 2,\n 'labels': {'app.kubernetes.io/component': 'llminferenceservice-workload',\n 'app.kubernetes.io/name': 'lora-multiple-adapters-test',\n 'app.kubernetes.io/part-of': 'llminferenceservice',\n 'kserve.io/component': 'workload',\n 'llm-d.ai/role': 'both',\n 'pod-template-hash': '77485bff95'},\n 'managed_fields': [{'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.ovn.org/pod-networks': {}}}},\n 'manager': 'ip-10-0-131-57',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:generateName': {},\n 'f:labels': {'.': {},\n 'f:app.kubernetes.io/component': [e2e-llm-inference-service] {},\n 'f:app.kubernetes.io/name': {},\n 'f:app.kubernetes.io/part-of': {},\n 'f:kserve.io/component': {},\n 'f:llm-d.ai/role': {},\n 'f:pod-template-hash': {}},\n 'f:ownerReferences': {'.': {},\n 'k:{"uid":"75c381c4-1742-4fb9-b05f-92ea6db35c9c"}': {}}},\n 'f:spec': {'f:containers': {'k:{"name":"main"}': {'.': {},\n 'f:args': {},\n 'f:command': {},\n 'f:env': {'.': {},\n 'k:{"name":"HF_HUB_CACHE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HOME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"TORCHINDUCTOR_CACHE_DIR"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"USER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_CPU_KVCACHE_SPACE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n [e2e-llm-inference-service] 'k:{"name":"VLLM_LOGGING_LEVEL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:lifecycle': {'.': {},\n 'f:preStop': {'.': {},\n 'f:exec': {'.': {},\n 'f:command': {}}}},\n 'f:livenessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:name': {},\n 'f:ports': {'.': {},\n 'k:{"containerPort":8000,"protocol":"TCP"}': {'.': {},\n 'f:containerPort': {},\n 'f:protocol': {}}},\n 'f:readinessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n [e2e-llm-inference-service] 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:securityContext': {'.': {},\n 'f:allowPrivilegeEscalation': {},\n 'f:capabilities': {'.': {},\n 'f:drop': {}},\n 'f:readOnlyRootFilesystem': {},\n 'f:runAsNonRoot': {},\n 'f:seccompProfile': {'.': {},\n 'f:type': {}}},\n 'f:startupProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:terminationMessagePath': {},\n [e2e-llm-inference-service] 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/dev/shm"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/home"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/models"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/tmp"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/var/run/kserve/tls"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}}}}},\n 'f:dnsPolicy': {},\n 'f:enableServiceLinks': {},\n 'f:initContainers': {'.': {},\n 'k:{"name":"storage-initializer"}': {'.': {},\n 'f:args': {},\n [e2e-llm-inference-service] 'f:env': {'.': {},\n 'k:{"name":"AWS_ACCESS_KEY_ID"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"AWS_CA_BUNDLE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_ENDPOINT_URL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_SECRET_ACCESS_KEY"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}': {'.': {},\n 'f:name': {},\n [e2e-llm-inference-service] 'f:value': {}},\n 'k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_HIGH_PERFORMANCE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_ENDPOINT"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_USE_HTTPS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_VERIFY_SSL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:name': {},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n [e2e-llm-inference-service] 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:terminationMessagePath': {},\n 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/etc/ssl/custom-certs"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}}}}},\n 'f:restartPolicy': {},\n 'f:schedulerName': {},\n 'f:securityContext': {},\n 'f:terminationGracePeriodSeconds': {},\n 'f:volumes': {'.': {},\n 'k:{"name":"cabundle-cert"}': {'.': {},\n 'f:configMap': {'.': {},\n 'f:defaultMode': {},\n 'f:name': {}},\n 'f:name': {}},\n 'k:{"name":"dshm"}': {'.': {},\n 'f:emptyDir': {'.': {},\n 'f:medium': {},\n [e2e-llm-inference-service] 'f:sizeLimit': {}},\n 'f:name': {}},\n 'k:{"name":"home"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"kserve-provision-location"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"model-cache"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"tls-certs"}': {'.': {},\n 'f:name': {},\n 'f:secret': {'.': {},\n 'f:defaultMode': {},\n 'f:secretName': {}}},\n 'k:{"name":"tmp-dir"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}}}}},\n 'manager': 'kube-controller-manager',\n 'operation': 'Update',\n 'subresource': None,\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.v1.cni.cncf.io/network-status': {}}}},\n 'manager': 'multus-daemon',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:status': {'f:conditions': {'k:{"type":"ContainersReady"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"Initialized"}': {'.': {},\n [e2e-llm-inference-service] 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodReadyToStartContainers"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodScheduled"}': {'f:observedGeneration': {}},\n 'k:{"type":"Ready"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}}},\n 'f:containerStatuses': {},\n 'f:hostIP': {},\n 'f:hostIPs': {},\n 'f:initContainerStatuses': {},\n 'f:observedGeneration': {},\n 'f:phase': {},\n 'f:podIP': {},\n 'f:podIPs': {'.': {},\n 'k:{"ip":"10.134.0.76"}': {'.': {},\n 'f:ip': {}}},\n 'f:startTime': {}}},\n 'manager': 'kubelet',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 51, 53, tzinfo=tzlocal())}],\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95-2tx5t',\n 'namespace': 'kserve-ci-e2e-test',\n 'owner_references': [{'api_version': 'apps/v1',\n 'block_owner_deletion': True,\n 'controller': True,\n 'kind': 'ReplicaSet',\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95',\n 'uid': '75c381c4-1742-4fb9-b05f-92ea6db35 [e2e-llm-inference-service] c9c'}],\n 'resource_version': '79769',\n 'self_link': None,\n 'uid': 'd4948f86-f0d1-4911-908f-7cb88db94468'},\n 'spec': {'active_deadline_seconds': None,\n 'affinity': None,\n 'automount_service_account_token': None,\n 'containers': [{'args': ['--enable-lora',\n '--max-lora-rank=64',\n '--max-loras=2',\n '--max-cpu-loras=4',\n '--lora-modules',\n '\'{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\''],\n 'command': ['/bin/bash',\n '-c',\n 'if [ -f /etc/profile.d/ibm-aiu-setup.sh '\n ']; then\n'\n ' source '\n '/etc/profile.d/ibm-aiu-setup.sh\n'\n 'fi\n'\n '\n'\n 'if [ "$KSERVE_INFER_ROCE" = "true" ]; '\n 'then\n'\n ' echo "Trying to infer RoCE configs '\n '... "\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gids/* '\n '2>/dev/null\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gid_attrs/types/* '\n '2>/dev/null\n'\n '\n'\n ' cat /proc/driver/nvidia/params\n'\n '\n'\n ' '\n 'KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE '\n 'v2"}\n'\n '\n'\n ' echo "[Infer RoCE] Discovering active '\n 'HCAs ..."\n'\n ' active_hcas=()\n'\n ' # Loop through all mlx5 devices found '\n 'in sysfs\n'\n ' for hca_dir in '\n '/sys/class/infiniband/mlx5_*; do\n'\n " # Ensure it's a directory before "\n 'proceeding\n'\n ' if [ -d "$hca_dir" ]; then\n'\n ' hca_name=$(basename '\n '"$hca_dir")\n'\n ' '\n 'port_state_file="$hca_dir/ports/1/state" '\n '# Assume port 1\n'\n ' '\n 'type_file="$hca_dir/ports/1/gid_attrs/types/*"\n'\n '\n'\n ' echo "[Infer RoCE] Check if '\n 'the port state file ${port_state_file} '\n 'exists and contains \'ACTIVE\'"\n'\n ' if [ -f "$port_state_file" ] '\n '&& grep -q "ACT [e2e-llm-inference-service] IVE" "$port_state_file" '\n '&& grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '${type_file} 2>/dev/null; then\n'\n ' echo "[Infer RoCE] Found '\n 'active HCA: $hca_name"\n'\n ' '\n 'active_hcas+=("$hca_name")\n'\n ' else\n'\n ' echo "[Infer RoCE] '\n 'Skipping inactive or down HCA: '\n '$hca_name"\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Check if we found any active HCAs\n'\n ' if [ ${#active_hcas[@]} -gt 0 ]; '\n 'then\n'\n ' # Join the array elements with a '\n 'comma\n'\n ' hca_port_pairs=()\n'\n ' for hca in "${active_hcas[@]}"; '\n 'do\n'\n ' hca_port_pairs+=("${hca}:1")\n'\n ' done\n'\n '\n'\n ' active_hca_list=$(IFS=,; echo '\n '"${active_hcas[*]}")\n'\n ' hca_port_pairs_list=$(IFS=,; echo '\n '"${hca_port_pairs[*]}")\n'\n ' echo "[Infer RoCE] Setting active '\n 'HCAs: ${active_hca_list}"\n'\n ' export '\n 'NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n'\n ' export '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n'\n ' export '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'NCCL_IB_HCA=${NCCL_IB_HCA}"\n'\n ' echo "[Infer RoCE] '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n'\n ' echo "[Infer RoCE] '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n'\n ' else\n'\n ' echo "[Infer RoCE] WARNING: No '\n 'active RoCE HCAs found. NCCL_IB_HCA '\n 'will not be set."\n'\n ' fi\n'\n '\n'\n ' if [ ${#active_hcas[@]} -gt 0 ]; '\n 'then\n'\n ' echo "[Infer RoCE] Finding '\n 'GID_INDEX for each active HCA (SR-IOV '\n 'compatible)..."\n'\n '\n'\n ' # For SR-IOV environments, find '\n 'the most common IPv4 RoCE v2 GID index '\n 'across all HCAs\n'\n ' declare -A gid_index_count\n'\n [e2e-llm-inference-service] ' declare -A hca_gid_index\n'\n '\n'\n ' for hca_name in '\n '"${active_hcas[@]}"; do\n'\n ' echo "[Infer RoCE] Processing '\n 'HCA: ${hca_name}"\n'\n '\n'\n ' # Find all RoCE v2 IPv4 GIDs '\n 'for this HCA and count by index\n'\n ' for tpath in '\n '/sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; '\n 'do\n'\n ' if grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '"$tpath" 2>/dev/null; then\n'\n ' idx=$(basename '\n '"$tpath")\n'\n ' '\n 'gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n'\n ' # Check for IPv4 GID '\n '(contains ffff:)\n'\n ' if [ -f "$gid_file" ] '\n '&& grep -q "ffff:" "$gid_file"; then\n'\n ' gid_value=$(cat '\n '"$gid_file" 2>/dev/null || echo "")\n'\n ' echo "[Infer '\n 'RoCE] Found IPv4 RoCE v2 GID for '\n '${hca_name}: index=${idx}, '\n 'gid=${gid_value}"\n'\n ' '\n 'hca_gid_index["${hca_name}"]="${idx}"\n'\n ' '\n 'gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} '\n '+ 1))\n'\n ' break # Use '\n 'first found IPv4 GID per HCA\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n ' done\n'\n '\n'\n ' # Find the most common GID index '\n '(most likely to be consistent across '\n 'nodes)\n'\n ' best_gid_index=""\n'\n ' max_count=0\n'\n ' for idx in '\n '"${!gid_index_count[@]}"; do\n'\n ' '\n 'count=${gid_index_count["${idx}"]}\n'\n ' echo "[Infer RoCE] GID_INDEX '\n '${idx} found on ${count} HCAs"\n'\n ' if [ $count -gt $max_count ]; '\n 'then\n'\n ' max_count=$count\n'\n ' best_gid_index="$idx"\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Use determinis [e2e-llm-inference-service] tic fallback if '\n 'tied - prefer index 3 (SR-IOV '\n 'standard)\n'\n ' if [ ${#gid_index_count[@]} -gt 1 '\n ']; then\n'\n ' echo "[Infer RoCE] Multiple '\n 'GID indices found, selecting most '\n 'common: ${best_gid_index}"\n'\n " # If there's a tie, prefer "\n "index 3 as it's most common in SR-IOV "\n 'setups\n'\n ' if [ -n '\n '"${gid_index_count[\'3\']}" ] && [ '\n '"${gid_index_count[\'3\']}" -eq '\n '"$max_count" ]; then\n'\n ' best_gid_index="3"\n'\n ' echo "[Infer RoCE] Using '\n 'deterministic fallback: GID_INDEX=3 '\n '(SR-IOV standard)"\n'\n ' fi\n'\n ' fi\n'\n '\n'\n ' # Check if GID_INDEX is already '\n 'set via environment variables\n'\n ' if [ -n "${NCCL_IB_GID_INDEX}" ]; '\n 'then\n'\n ' echo "[Infer RoCE] Using '\n 'pre-configured '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} '\n 'from environment"\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' echo "[Infer RoCE] Using '\n 'pre-configured '\n 'GID_INDEX=${NCCL_IB_GID_INDEX} for '\n 'NCCL, NVSHMEM, and UCX"\n'\n ' elif [ -n "$best_gid_index" ]; '\n 'then\n'\n ' echo "[Infer RoCE] Selected '\n 'GID_INDEX: ${best_gid_index} (found on '\n '${max_count} HCAs)"\n'\n '\n'\n ' export '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n'\n '\n'\n ' echo "[Infer RoCE] Exported '\n 'GID_INDEX=${best_gid_index} for NCCL, '\n 'NVSHMEM, and UCX"\n'\n ' else\n'\n ' echo "[Infer RoCE] ERROR: No '\n 'valid IPv4 '\n '${KSERVE_INFER_IB_GID_INDEX_GREP} '\n 'GID_INDEX found on any HCA."\n'\n [e2e-llm-inference-service] ' fi\n'\n ' else\n'\n ' echo "[Infer RoCE] No active HCAs '\n 'found, skipping GID_INDEX inference."\n'\n ' fi\n'\n 'fi\n'\n '\n'\n '# --disable-access-log-for-endpoints '\n 'landed in vLLM 0.16.0 '\n '(vllm-project/vllm#30011).\n'\n '# Older versions still need the blanket '\n '--disable-uvicorn-access-log.\n'\n 'ACCESS_LOG_ARGS="--disable-uvicorn-access-log"\n'\n 'VLLM_VERSION=$(vllm --version '\n "2>/dev/null | tail -1 | awk '{print "\n "$NF}')\n"\n 'echo "[access-log-detect] vllm '\n 'version=\'${VLLM_VERSION}\'"\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ "$(printf '\n '\'%s\\n%s\\n\' "0.16.0" '\n '"${VLLM_VERSION}" | sort -V | head -1)" '\n '= "0.16.0" ]; then\n'\n ' '\n 'ACCESS_LOG_ARGS="--disable-access-log-for-endpoints '\n '/health,/metrics,/ping"\n'\n 'fi\n'\n 'echo "[access-log-detect] selected '\n 'ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n'\n '\n'\n '# --shutdown-timeout landed in vLLM '\n '0.18.0 (vllm-project/vllm#36666).\n'\n 'SHUTDOWN_TIMEOUT_ARGS=""\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ "$(printf '\n '\'%s\\n%s\\n\' "0.18.0" '\n '"${VLLM_VERSION}" | sort -V | head -1)" '\n '= "0.18.0" ]; then\n'\n ' '\n 'SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout '\n '40"\n'\n 'fi\n'\n '\n'\n 'eval "exec vllm serve /mnt/models \\\n'\n ' --served-model-name '\n '"facebook/opt-125m" '\n '"publishers/kserve-ci-e2e-test/models/facebook/opt-125m" '\n '\\\n'\n ' --port 8000 \\\n'\n ' ${ACCESS_LOG_ARGS} \\\n'\n ' ${SHUTDOWN_TIMEOUT_ARGS} \\\n'\n ' --enable-ssl-refresh \\\n'\n ' --ssl-certfile '\n '/var/run/kserve/tls/tls.crt \\\n'\n ' --ssl-keyfile '\n '/var/run/kserve/tls/tls.key \\\n'\n ' ${VLLM_ADDITIONAL_ARGS} \\\n'\n ' $@"',\n '--'],\n 'env': [{'name': 'HOME',\n 'value': '/home',\n 'value_from': None},\n {'name': 'VLLM [e2e-llm-inference-service] _LOGGING_LEVEL',\n 'value': 'DEBUG',\n 'value_from': None},\n {'name': 'VLLM_CPU_KVCACHE_SPACE',\n 'value': '1',\n 'value_from': None},\n {'name': 'VLLM_ENABLE_V1_MULTIPROCESSING',\n 'value': '0',\n 'value_from': None},\n {'name': 'USER',\n 'value': 'nonroot',\n 'value_from': None},\n {'name': 'TORCHINDUCTOR_CACHE_DIR',\n 'value': '/tmp/torchinductor-cache',\n 'value_from': None},\n {'name': 'HF_HUB_CACHE',\n 'value': '/models',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': {'post_start': None,\n 'pre_stop': {'_exec': {'command': ['/bin/sleep',\n '15']},\n 'http_get': None,\n 'sleep': None,\n 'tcp_socket': None}},\n 'liveness_probe': {'_exec': None,\n 'failure_threshold': 10,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'name': 'main',\n 'ports': [{'container_port': 8000,\n 'host_ip': None,\n 'host_port': None,\n 'name': None,\n 'protocol': 'TCP'}],\n 'readiness_probe': {'_exec': None,\n 'failure_threshold': 2,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 1,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'resize_policy': [e2e-llm-inference-service] None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2', 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n 'read_only_root_filesystem': True,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'windows_options': None},\n 'startup_probe': {'_exec': None,\n 'failure_threshold': 60,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/home',\n 'mount_propagation': None,\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/tmp',\n 'mount_propagation': None,\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/dev/shm',\n 'mount_propagation': None,\n 'name': 'dshm',\n 'read_only': None,\n 'recursive_read_only': None,\n [e2e-llm-inference-service] 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/models',\n 'mount_propagation': None,\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/kserve/tls',\n 'mount_propagation': None,\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'dns_config': None,\n 'dns_policy': 'ClusterFirst',\n 'enable_service_links': True,\n 'ephemeral_containers': None,\n 'host_aliases': None,\n 'host_ipc': None,\n 'host_network': None,\n 'host_pid': None,\n 'host_users': None,\n 'hostname': None,\n 'image_pull_secrets': [{'name': 'default-dockercfg-6qcsh'}],\n 'init_containers': [{'args': ['hf://facebook/opt-125m',\n '/mnt/models',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-1',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-2'],\n 'command': None,\n 'env': [{'name': 'AWS_ACCESS_KEY_ID',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': None,\n 'secret_key_ref': {'key': 'AWS_ACCESS_KEY_ID',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'AWS_SECRET_ACCESS_KEY',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': N [e2e-llm-inference-service] one,\n 'secret_key_ref': {'key': 'AWS_SECRET_ACCESS_KEY',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'S3_USE_HTTPS',\n 'value': '0',\n 'value_from': None},\n {'name': 'S3_ENDPOINT',\n 'value': 's3-service.kserve:8333',\n 'value_from': None},\n {'name': 'AWS_ENDPOINT_URL',\n 'value': 'http://s3-service.kserve:8333',\n 'value_from': None},\n {'name': 'S3_VERIFY_SSL',\n 'value': '0',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE',\n 'value': '/etc/ssl/custom-certs/cabundle.crt',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE_CONFIGMAP',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'HF_HUB_ENABLE_HF_TRANSFER',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_HIGH_PERFORMANCE',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_NUM_CONCURRENT_RANGE_GETS',\n 'value': '8',\n 'value_from': None},\n {'name': 'CA_BUNDLE_CONFIGMAP_NAME',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'CA_BUNDLE_VOLUME_MOUNT_POINT',\n 'value': '/etc/ssl/custom-certs',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': None,\n 'liveness_probe': None,\n 'name': 'storage-initializer',\n 'ports': None,\n 'readiness_probe': None,\n 'resize_policy': None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n [e2e-llm-inference-service] 'read_only_root_filesystem': None,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': None,\n 'windows_options': None},\n 'startup_probe': None,\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'mount_propagation': None,\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'node_name': 'ip-10-0-131-57.ec2.internal',\n 'node_selector': None,\n 'os': None,\n 'overhead': None,\n 'preemption_policy': 'PreemptLowerPriority',\n 'priority': 0,\n 'priority_class_name': None,\n 'readiness_gates': None,\n 'resource_claims': None,\n 'resources': None,\n 'restart_policy': 'Always',\n 'runtime_class_name': None,\n 'scheduler_name': 'default-scheduler',\n 'scheduling_gates': None,\n 'security_context': {'app_armor_profile': None,\n 'fs_group': 1000690000,\n 'fs_group_change_policy': None,\n 'run_as_group': None,\n 'run_as_non_root': None,\n 'run_as_user': None,\n 'se_linux_change_policy': None,\n 'se_linux_options': {'level': 's0:c26,c20',\n 'role': None,\n 'type': None,\n 'user': None},\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'supplemental_groups': None,\n [e2e-llm-inference-service] 'supplemental_groups_policy': None,\n 'sysctls': None,\n 'windows_options': None},\n 'service_account': 'default',\n 'service_account_name': 'default',\n 'set_hostname_as_fqdn': None,\n 'share_process_namespace': None,\n 'subdomain': None,\n 'termination_grace_period_seconds': 60,\n 'tolerations': [{'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/not-ready',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/unreachable',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoSchedule',\n 'key': 'node.kubernetes.io/memory-pressure',\n 'operator': 'Exists',\n 'toleration_seconds': None,\n 'value': None}],\n 'topology_spread_constraints': None,\n 'volumes': [{'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'home',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': 'Memory', 'size_limit': '1Gi'},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'dshm',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_b [e2e-llm-inference-service] lock_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'model-cache',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tmp-dir',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tls-certs',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': {'default_mode': 420,\n 'items': None,\n [e2e-llm-inference-service] 'optional': None,\n 'secret_name': 'lora-multiple-adapters-test-kserve-self-signed-certs'},\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None, 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kserve-provision-location',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': {'default_mode': 420,\n 'items': None,\n 'name': 'odh-kserve-custom-ca-bundle',\n 'optional': None},\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'cabundle-cert',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kube-api-access-n5dws',\n [e2e-llm-inference-service] 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': {'default_mode': 420,\n 'sources': [{'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': {'audience': None,\n 'expiration_seconds': 3607,\n 'path': 'token'}},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'ca.crt',\n 'mode': None,\n 'path': 'ca.crt'}],\n 'name': 'kube-root-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': {'items': [{'field_ref': {'api_version': 'v1',\n 'field_path': 'metadata.namespace'},\n 'mode': None,\n 'path': 'namespace',\n 'resource_field_ref': None}]},\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'service-ca.crt',\n 'mode': None,\n 'path': 'service-ca.crt'}],\n 'name': 'openshift-service-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None}]},\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None}]},\n 'status': {'conditions': [{'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 18, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodReadyToStartContainers'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal()),\n 'message': None,\n [e2e-llm-inference-service] 'reason': None,\n 'status': 'True',\n 'type': 'Initialized'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'Ready'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'ContainersReady'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodScheduled'}],\n 'container_statuses': [{'allocated_resources': {'cpu': '200m',\n 'memory': '2Gi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402',\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_id': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'main',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2',\n 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_count': 0,\n 'started': True,\n 'state': {'running': {'started_at': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal())},\n 'terminated': None,\n 'waiting': None},\n 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/home',\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/tmp',\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/dev/shm',\n 'name': 'dshm',\n 'read_only': None,\n [e2e-llm-inference-service] 'recursive_read_only': None},\n {'mount_path': '/models',\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/var/run/kserve/tls',\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'ephemeral_container_statuses': None,\n 'host_i_ps': [{'ip': '10.0.131.57'}],\n 'host_ip': '10.0.131.57',\n 'init_container_statuses': [{'allocated_resources': {'cpu': '100m',\n 'memory': '100Mi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_id': 'quay.io/opendatahub/kserve-storage-initializer@sha256:60e2a5c8616d82c8933b614ae003dd3412a420a29a0be570b6a6439a3d39ab99',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'storage-initializer',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_count': 0,\n 'started': False,\n 'state': {'running': None,\n 'terminated': {'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'exit_code': 0,\n 'finished_at': datetime.datetime(2026, 7, 6, 21, 35, 22, tzinfo=tzlocal()),\n 'message': None,\n 'reason': 'Completed',\n 'signal': None,\n 'started_at': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n [e2e-llm-inference-service] 'waiting': None},\n 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'message': None,\n 'nominated_node_name': None,\n 'phase': 'Running',\n 'pod_i_ps': [{'ip': '10.134.0.76'}],\n 'pod_ip': '10.134.0.76',\n 'qos_class': 'Burstable',\n 'reason': None,\n 'resize': None,\n 'resource_claim_statuses': None,\n 'start_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())}}] = {'api_version': 'v1',\n 'items': [{'api_version': None,\n 'kind': None,\n 'metadata': {'annotations': {'k8s.ovn.org/pod-networks': '{"default":{"ip_addresses":["10.134.0.76/23"],"mac_address":"0a:58:0a:86:00:4c","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.76/23","gateway_ip":"10.134.0.1","role":"primary"}}',\n 'k8s.v1.cni.cncf.io/network-status': '[{\n'\n ' '\n '"name": '\n '"ovn-kubernetes",\n'\n ' '\n '"interface": '\n '"eth0",\n'\n ' '\n '"ips": '\n '[\n'\n ' '\n '"10.134.0.76"\n'\n ' '\n '],\n'\n ' '\n '"mac": '\n '"0a:58:0a:86:00:4c",\n'\n ' '\n [e2e-llm-inference-service] '"default": '\n 'true,\n'\n ' '\n '"dns": '\n '{}\n'\n '}]',\n 'openshift.io/scc': 'restricted-v2',\n 'seccomp.security.alpha.kubernetes.io/pod': 'runtime/default',\n 'security.openshift.io/validated-scc-subject-type': 'user'},\n 'creation_timestamp': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal()),\n 'deletion_grace_period_seconds': 60,\n 'deletion_timestamp': datetime.datetime(2026, 7, 6, 21, 52, 53, tzinfo=tzlocal()),\n 'finalizers': None,\n 'generate_name': 'lora-multiple-adapters-test-kserve-77485bff95-',\n 'generation': 2,\n 'labels': {'app.kubernetes.io/component': 'llminferenceservice-workload',\n 'app.kubernetes.io/name': 'lora-multiple-adapters-test',\n 'app.kubernetes.io/part-of': 'llminferenceservice',\n 'kserve.io/component': 'workload',\n 'llm-d.ai/role': 'both',\n 'pod-template-hash': '77485bff95'},\n 'managed_fields': [{'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.ovn.org/pod-networks': {}}}},\n 'manager': 'ip-10-0-131-57',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:generateName': {},\n 'f:labels': {'.': {},\n 'f:app.kubernetes.io/component': {},\n 'f:app.kubernetes.io/name': {},\n 'f:app.kubernetes.io/part-of': {},\n 'f:kserve.io/component': {},\n 'f:llm-d.ai/role': {},\n 'f:pod-template-hash': {}},\n 'f:ownerReferences': {'.': {},\n 'k:{"uid":"75c381c4-1742-4fb9-b05f-92ea6db35c9c"}': {}}},\n 'f:spec': {'f:containers': {'k:{"name":"main"}': {'.': {},\n 'f:args': {},\n 'f:command': {},\n [e2e-llm-inference-service] 'f:env': {'.': {},\n 'k:{"name":"HF_HUB_CACHE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HOME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"TORCHINDUCTOR_CACHE_DIR"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"USER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_CPU_KVCACHE_SPACE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"VLLM_LOGGING_LEVEL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n 'f:lifecycle': {'.': {},\n 'f:pr [e2e-llm-inference-service] eStop': {'.': {},\n 'f:exec': {'.': {},\n 'f:command': {}}}},\n 'f:livenessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:name': {},\n 'f:ports': {'.': {},\n 'k:{"containerPort":8000,"protocol":"TCP"}': {'.': {},\n 'f:containerPort': {},\n 'f:protocol': {}}},\n 'f:readinessProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:resources': {'.': {},\n [e2e-llm-inference-service] 'f:limits': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:securityContext': {'.': {},\n 'f:allowPrivilegeEscalation': {},\n 'f:capabilities': {'.': {},\n 'f:drop': {}},\n 'f:readOnlyRootFilesystem': {},\n 'f:runAsNonRoot': {},\n 'f:seccompProfile': {'.': {},\n 'f:type': {}}},\n 'f:startupProbe': {'.': {},\n 'f:failureThreshold': {},\n 'f:httpGet': {'.': {},\n 'f:path': {},\n 'f:port': {},\n 'f:scheme': {}},\n 'f:periodSeconds': {},\n 'f:successThreshold': {},\n 'f:timeoutSeconds': {}},\n 'f:terminationMessagePath': {},\n 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n [e2e-llm-inference-service] 'k:{"mountPath":"/dev/shm"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/home"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/models"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/tmp"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}},\n 'k:{"mountPath":"/var/run/kserve/tls"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}}}}},\n 'f:dnsPolicy': {},\n 'f:enableServiceLinks': {},\n 'f:initContainers': {'.': {},\n 'k:{"name":"storage-initializer"}': {'.': {},\n 'f:args': {},\n [e2e-llm-inference-service] 'f:env': {'.': {},\n 'k:{"name":"AWS_ACCESS_KEY_ID"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"AWS_CA_BUNDLE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_ENDPOINT_URL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"AWS_SECRET_ACCESS_KEY"}': {'.': {},\n 'f:name': {},\n 'f:valueFrom': {'.': {},\n 'f:secretKeyRef': {}}},\n 'k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}': {'.': {},\n [e2e-llm-inference-service] 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_HIGH_PERFORMANCE"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_ENDPOINT"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_USE_HTTPS"}': {'.': {},\n 'f:name': {},\n 'f:value': {}},\n 'k:{"name":"S3_VERIFY_SSL"}': {'.': {},\n 'f:name': {},\n 'f:value': {}}},\n 'f:image': {},\n 'f:imagePullPolicy': {},\n [e2e-llm-inference-service] 'f:name': {},\n 'f:resources': {'.': {},\n 'f:limits': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}},\n 'f:requests': {'.': {},\n 'f:cpu': {},\n 'f:memory': {}}},\n 'f:terminationMessagePath': {},\n 'f:terminationMessagePolicy': {},\n 'f:volumeMounts': {'.': {},\n 'k:{"mountPath":"/etc/ssl/custom-certs"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {},\n 'f:readOnly': {}},\n 'k:{"mountPath":"/mnt"}': {'.': {},\n 'f:mountPath': {},\n 'f:name': {}}}}},\n 'f:restartPolicy': {},\n 'f:schedulerName': {},\n 'f:securityContext': {},\n 'f:terminationGracePeriodSeconds': {},\n 'f:volumes': {'.': {},\n 'k:{"name":"cabundle-cert"}': {'.': {},\n 'f:configMap': {'.': {},\n 'f:defaultMode': {} [e2e-llm-inference-service] ,\n 'f:name': {}},\n 'f:name': {}},\n 'k:{"name":"dshm"}': {'.': {},\n 'f:emptyDir': {'.': {},\n 'f:medium': {},\n 'f:sizeLimit': {}},\n 'f:name': {}},\n 'k:{"name":"home"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"kserve-provision-location"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"model-cache"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}},\n 'k:{"name":"tls-certs"}': {'.': {},\n 'f:name': {},\n 'f:secret': {'.': {},\n 'f:defaultMode': {},\n 'f:secretName': {}}},\n 'k:{"name":"tmp-dir"}': {'.': {},\n 'f:emptyDir': {},\n 'f:name': {}}}}},\n 'manager': 'kube-controller-manager',\n 'operation': 'Update',\n 'subresource': None,\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:metadata': {'f:annotations': {'f:k8s.v1.cni.cncf.io/network-status': {}}}},\n 'manager': 'multus-daemon',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\ [e2e-llm-inference-service] n {'api_version': 'v1',\n 'fields_type': 'FieldsV1',\n 'fields_v1': {'f:status': {'f:conditions': {'k:{"type":"ContainersReady"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"Initialized"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodReadyToStartContainers"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}},\n 'k:{"type":"PodScheduled"}': {'f:observedGeneration': {}},\n 'k:{"type":"Ready"}': {'.': {},\n 'f:lastProbeTime': {},\n 'f:lastTransitionTime': {},\n 'f:observedGeneration': {},\n 'f:status': {},\n 'f:type': {}}},\n 'f:containerStatuses': {},\n 'f:hostIP': {},\n 'f:hostIPs': {},\n 'f:initContainerStatuses': {},\n [e2e-llm-inference-service] 'f:observedGeneration': {},\n 'f:phase': {},\n 'f:podIP': {},\n 'f:podIPs': {'.': {},\n 'k:{"ip":"10.134.0.76"}': {'.': {},\n 'f:ip': {}}},\n 'f:startTime': {}}},\n 'manager': 'kubelet',\n 'operation': 'Update',\n 'subresource': 'status',\n 'time': datetime.datetime(2026, 7, 6, 21, 51, 53, tzinfo=tzlocal())}],\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95-2tx5t',\n 'namespace': 'kserve-ci-e2e-test',\n 'owner_references': [{'api_version': 'apps/v1',\n 'block_owner_deletion': True,\n 'controller': True,\n 'kind': 'ReplicaSet',\n 'name': 'lora-multiple-adapters-test-kserve-77485bff95',\n 'uid': '75c381c4-1742-4fb9-b05f-92ea6db35c9c'}],\n 'resource_version': '79769',\n 'self_link': None,\n 'uid': 'd4948f86-f0d1-4911-908f-7cb88db94468'},\n 'spec': {'active_deadline_seconds': None,\n 'affinity': None,\n 'automount_service_account_token': None,\n 'containers': [{'args': ['--enable-lora',\n '--max-lora-rank=64',\n '--max-loras=2',\n '--max-cpu-loras=4',\n '--lora-modules',\n '\'{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}\'',\n '\'{"name":"lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\'',\n '\'{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-2","path":"/mnt/lora/lora-adapter-2"}\''],\n 'command': ['/bin/bash',\n '-c',\n 'if [ -f '\n '/etc/profile.d/ibm-aiu-setup.sh '\n ']; then\n'\n ' source '\n '/etc/profile.d/ibm-aiu-setup.sh\n'\n 'fi\n'\n '\n'\n 'if [ "$KSERVE_INFER_ROCE" = '\n '"true" ]; then\n'\n ' echo "Trying to infer RoCE '\n 'configs ... "\n'\n ' grep -H . '\n '/sys/class/infiniband/*/ports/*/gids/* '\n '2>/dev/null\n'\n ' grep -H . '\n '/sys/class/inf [e2e-llm-inference-service] iniband/*/ports/*/gid_attrs/types/* '\n '2>/dev/null\n'\n '\n'\n ' cat '\n '/proc/driver/nvidia/params\n'\n '\n'\n ' '\n 'KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-"RoCE '\n 'v2"}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'Discovering active HCAs '\n '..."\n'\n ' active_hcas=()\n'\n ' # Loop through all mlx5 '\n 'devices found in sysfs\n'\n ' for hca_dir in '\n '/sys/class/infiniband/mlx5_*; '\n 'do\n'\n " # Ensure it's a "\n 'directory before proceeding\n'\n ' if [ -d "$hca_dir" ]; '\n 'then\n'\n ' '\n 'hca_name=$(basename '\n '"$hca_dir")\n'\n ' '\n 'port_state_file="$hca_dir/ports/1/state" '\n '# Assume port 1\n'\n ' '\n 'type_file="$hca_dir/ports/1/gid_attrs/types/*"\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'Check if the port state file '\n '${port_state_file} exists '\n 'and contains \'ACTIVE\'"\n'\n ' if [ -f '\n '"$port_state_file" ] && grep '\n '-q "ACTIVE" '\n '"$port_state_file" && grep '\n '-q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '${type_file} 2>/dev/null; '\n 'then\n'\n ' echo "[Infer '\n 'RoCE] Found active HCA: '\n '$hca_name"\n'\n ' '\n 'active_hcas+=("$hca_name")\n'\n ' else\n'\n ' echo "[Infer '\n 'RoCE] Skipping inactive or '\n 'down HCA: $hca_name"\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n '\n'\n [e2e-llm-inference-service] ' # Check if we found any '\n 'active HCAs\n'\n ' if [ ${#active_hcas[@]} '\n '-gt 0 ]; then\n'\n ' # Join the array '\n 'elements with a comma\n'\n ' hca_port_pairs=()\n'\n ' for hca in '\n '"${active_hcas[@]}"; do\n'\n ' '\n 'hca_port_pairs+=("${hca}:1")\n'\n ' done\n'\n '\n'\n ' '\n 'active_hca_list=$(IFS=,; '\n 'echo "${active_hcas[*]}")\n'\n ' '\n 'hca_port_pairs_list=$(IFS=,; '\n 'echo '\n '"${hca_port_pairs[*]}")\n'\n ' echo "[Infer RoCE] '\n 'Setting active HCAs: '\n '${active_hca_list}"\n'\n ' export '\n 'NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n'\n ' export '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n'\n ' export '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'NCCL_IB_HCA=${NCCL_IB_HCA}"\n'\n ' echo "[Infer RoCE] '\n 'NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}"\n'\n ' echo "[Infer RoCE] '\n 'UCX_NET_DEVICES=${UCX_NET_DEVICES}"\n'\n ' else\n'\n ' echo "[Infer RoCE] '\n 'WARNING: No active RoCE HCAs '\n 'found. NCCL_IB_HCA will not '\n 'be set."\n'\n ' fi\n'\n '\n'\n ' if [ ${#active_hcas[@]} '\n '-gt 0 ]; then\n'\n ' echo "[Infer RoCE] '\n 'Finding GID_INDEX for each '\n 'active HCA (SR-IOV '\n 'compatible)..."\n'\n '\n'\n ' # For SR-IOV '\n 'environments, find the most '\n 'common IPv4 RoCE v2 GID '\n 'index across all HCAs\n'\n ' declare -A '\n [e2e-llm-inference-service] 'gid_index_count\n'\n ' declare -A '\n 'hca_gid_index\n'\n '\n'\n ' for hca_name in '\n '"${active_hcas[@]}"; do\n'\n ' echo "[Infer RoCE] '\n 'Processing HCA: '\n '${hca_name}"\n'\n '\n'\n ' # Find all RoCE v2 '\n 'IPv4 GIDs for this HCA and '\n 'count by index\n'\n ' for tpath in '\n '/sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*; '\n 'do\n'\n ' if grep -q '\n '"${KSERVE_INFER_IB_GID_INDEX_GREP}" '\n '"$tpath" 2>/dev/null; then\n'\n ' '\n 'idx=$(basename "$tpath")\n'\n ' '\n 'gid_file="/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}"\n'\n ' # Check '\n 'for IPv4 GID (contains '\n 'ffff:)\n'\n ' if [ -f '\n '"$gid_file" ] && grep -q '\n '"ffff:" "$gid_file"; then\n'\n ' '\n 'gid_value=$(cat "$gid_file" '\n '2>/dev/null || echo "")\n'\n ' echo '\n '"[Infer RoCE] Found IPv4 '\n 'RoCE v2 GID for ${hca_name}: '\n 'index=${idx}, '\n 'gid=${gid_value}"\n'\n ' '\n 'hca_gid_index["${hca_name}"]="${idx}"\n'\n ' '\n 'gid_index_count["${idx}"]=$((${gid_index_count["${idx}"]} '\n '+ 1))\n'\n ' break '\n '# Use first found IPv4 GID '\n 'per HCA\n'\n ' fi\n'\n ' fi\n'\n ' done\n'\n ' done\n'\n '\n'\n ' # Find the most common '\n 'GID index (most likely to be '\n 'consistent across nodes)\n'\n ' best_gid_index=""\n'\n [e2e-llm-inference-service] ' max_count=0\n'\n ' for idx in '\n '"${!gid_index_count[@]}"; '\n 'do\n'\n ' '\n 'count=${gid_index_count["${idx}"]}\n'\n ' echo "[Infer RoCE] '\n 'GID_INDEX ${idx} found on '\n '${count} HCAs"\n'\n ' if [ $count -gt '\n '$max_count ]; then\n'\n ' '\n 'max_count=$count\n'\n ' '\n 'best_gid_index="$idx"\n'\n ' fi\n'\n ' done\n'\n '\n'\n ' # Use deterministic '\n 'fallback if tied - prefer '\n 'index 3 (SR-IOV standard)\n'\n ' if [ '\n '${#gid_index_count[@]} -gt 1 '\n ']; then\n'\n ' echo "[Infer RoCE] '\n 'Multiple GID indices found, '\n 'selecting most common: '\n '${best_gid_index}"\n'\n " # If there's a "\n "tie, prefer index 3 as it's "\n 'most common in SR-IOV '\n 'setups\n'\n ' if [ -n '\n '"${gid_index_count[\'3\']}" '\n '] && [ '\n '"${gid_index_count[\'3\']}" '\n '-eq "$max_count" ]; then\n'\n ' '\n 'best_gid_index="3"\n'\n ' echo "[Infer '\n 'RoCE] Using deterministic '\n 'fallback: GID_INDEX=3 '\n '(SR-IOV standard)"\n'\n ' fi\n'\n ' fi\n'\n '\n'\n ' # Check if GID_INDEX '\n 'is already set via '\n 'environment variables\n'\n ' if [ -n '\n '"${NCCL_IB_GID_INDEX}" ]; '\n 'then\n'\n ' echo "[Infer RoCE] '\n 'Using pre-configured '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX} '\n 'from environment"\n'\n [e2e-llm-inference-service] ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n'\n ' echo "[Infer RoCE] '\n 'Using pre-configured '\n 'GID_INDEX=${NCCL_IB_GID_INDEX} '\n 'for NCCL, NVSHMEM, and UCX"\n'\n ' elif [ -n '\n '"$best_gid_index" ]; then\n'\n ' echo "[Infer RoCE] '\n 'Selected GID_INDEX: '\n '${best_gid_index} (found on '\n '${max_count} HCAs)"\n'\n '\n'\n ' export '\n 'NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n'\n ' export '\n 'UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n'\n '\n'\n ' echo "[Infer RoCE] '\n 'Exported '\n 'GID_INDEX=${best_gid_index} '\n 'for NCCL, NVSHMEM, and UCX"\n'\n ' else\n'\n ' echo "[Infer RoCE] '\n 'ERROR: No valid IPv4 '\n '${KSERVE_INFER_IB_GID_INDEX_GREP} '\n 'GID_INDEX found on any '\n 'HCA."\n'\n ' fi\n'\n ' else\n'\n ' echo "[Infer RoCE] No '\n 'active HCAs found, skipping '\n 'GID_INDEX inference."\n'\n ' fi\n'\n 'fi\n'\n '\n'\n '# '\n '--disable-access-log-for-endpoints '\n 'landed in vLLM 0.16.0 '\n '(vllm-project/vllm#30011).\n'\n '# Older versions still need '\n 'the blanket '\n '--disable-uvicorn-access-log.\n'\n 'ACCESS_LOG_ARGS="--disable-uvicorn-access-log"\n'\n 'VLLM_VERSION=$(vllm '\n '--version 2>/dev/null | tail '\n "-1 | awk '{print $NF}')\n"\n 'echo "[access-log-detect] '\n 'vllm '\n [e2e-llm-inference-service] 'version=\'${VLLM_VERSION}\'"\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ '\n '"$(printf \'%s\\n%s\\n\' '\n '"0.16.0" "${VLLM_VERSION}" | '\n 'sort -V | head -1)" = '\n '"0.16.0" ]; then\n'\n ' '\n 'ACCESS_LOG_ARGS="--disable-access-log-for-endpoints '\n '/health,/metrics,/ping"\n'\n 'fi\n'\n 'echo "[access-log-detect] '\n 'selected '\n 'ACCESS_LOG_ARGS=\'${ACCESS_LOG_ARGS}\'"\n'\n '\n'\n '# --shutdown-timeout landed '\n 'in vLLM 0.18.0 '\n '(vllm-project/vllm#36666).\n'\n 'SHUTDOWN_TIMEOUT_ARGS=""\n'\n 'if [[ "$VLLM_VERSION" =~ '\n '^[0-9]+\\.[0-9]+ ]] && [ '\n '"$(printf \'%s\\n%s\\n\' '\n '"0.18.0" "${VLLM_VERSION}" | '\n 'sort -V | head -1)" = '\n '"0.18.0" ]; then\n'\n ' '\n 'SHUTDOWN_TIMEOUT_ARGS="--shutdown-timeout '\n '40"\n'\n 'fi\n'\n '\n'\n 'eval "exec vllm serve '\n '/mnt/models \\\n'\n ' --served-model-name '\n '"facebook/opt-125m" '\n '"publishers/kserve-ci-e2e-test/models/facebook/opt-125m" '\n '\\\n'\n ' --port 8000 \\\n'\n ' ${ACCESS_LOG_ARGS} \\\n'\n ' ${SHUTDOWN_TIMEOUT_ARGS} '\n '\\\n'\n ' --enable-ssl-refresh \\\n'\n ' --ssl-certfile '\n '/var/run/kserve/tls/tls.crt '\n '\\\n'\n ' --ssl-keyfile '\n '/var/run/kserve/tls/tls.key '\n '\\\n'\n ' ${VLLM_ADDITIONAL_ARGS} '\n '\\\n'\n ' $@"',\n '--'],\n 'env': [{'name': 'HOME',\n 'value': '/home',\n 'value_from': None},\n {'name': 'VLLM_LOGGING_LEVEL',\n 'value': 'DEBUG',\n 'value_from': None},\n [e2e-llm-inference-service] {'name': 'VLLM_CPU_KVCACHE_SPACE',\n 'value': '1',\n 'value_from': None},\n {'name': 'VLLM_ENABLE_V1_MULTIPROCESSING',\n 'value': '0',\n 'value_from': None},\n {'name': 'USER',\n 'value': 'nonroot',\n 'value_from': None},\n {'name': 'TORCHINDUCTOR_CACHE_DIR',\n 'value': '/tmp/torchinductor-cache',\n 'value_from': None},\n {'name': 'HF_HUB_CACHE',\n 'value': '/models',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': {'post_start': None,\n 'pre_stop': {'_exec': {'command': ['/bin/sleep',\n '15']},\n 'http_get': None,\n 'sleep': None,\n 'tcp_socket': None}},\n 'liveness_probe': {'_exec': None,\n 'failure_threshold': 10,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'name': 'main',\n 'ports': [{'container_port': 8000,\n 'host_ip': None,\n 'host_port': None,\n 'name': None,\n 'protocol': 'TCP'}],\n 'readiness_probe': {'_exec': None,\n 'failure_threshold': 2,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n [e2e-llm-inference-service] 'initial_delay_seconds': None,\n 'period_seconds': 1,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'resize_policy': None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2',\n 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n 'read_only_root_filesystem': True,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'windows_options': None},\n 'startup_probe': {'_exec': None,\n 'failure_threshold': 60,\n 'grpc': None,\n 'http_get': {'host': None,\n 'http_headers': None,\n 'path': '/health',\n 'port': 8000,\n 'scheme': 'HTTPS'},\n 'initial_delay_seconds': None,\n 'period_seconds': 10,\n 'success_threshold': 1,\n 'tcp_socket': None,\n 'termination_grace_period_seconds': None,\n 'timeout_seconds': 1},\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/home',\n 'mount_propagation': None,\n 'name': 'home',\n 'read_only': None,\n [e2e-llm-inference-service] 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/tmp',\n 'mount_propagation': None,\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/dev/shm',\n 'mount_propagation': None,\n 'name': 'dshm',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/models',\n 'mount_propagation': None,\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/kserve/tls',\n 'mount_propagation': None,\n 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'dns_config': None,\n 'dns_policy': 'ClusterFirst',\n 'enable_service_links': True,\n 'ephemeral_containers': None,\n 'host_aliases': None,\n 'host_ipc': None,\n 'host_network': None,\n 'host_pid' [e2e-llm-inference-service] : None,\n 'host_users': None,\n 'hostname': None,\n 'image_pull_secrets': [{'name': 'default-dockercfg-6qcsh'}],\n 'init_containers': [{'args': ['hf://facebook/opt-125m',\n '/mnt/models',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-1',\n 'hf://edbeeching/opt-125m-lora',\n '/mnt/lora/lora-adapter-2'],\n 'command': None,\n 'env': [{'name': 'AWS_ACCESS_KEY_ID',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': None,\n 'secret_key_ref': {'key': 'AWS_ACCESS_KEY_ID',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'AWS_SECRET_ACCESS_KEY',\n 'value': None,\n 'value_from': {'config_map_key_ref': None,\n 'field_ref': None,\n 'resource_field_ref': None,\n 'secret_key_ref': {'key': 'AWS_SECRET_ACCESS_KEY',\n 'name': 'seaweedfs-s3-creds',\n 'optional': None}}},\n {'name': 'S3_USE_HTTPS',\n 'value': '0',\n 'value_from': None},\n {'name': 'S3_ENDPOINT',\n 'value': 's3-service.kserve:8333',\n 'value_from': None},\n {'name': 'AWS_ENDPOINT_URL',\n 'value': 'http://s3-service.kserve:8333',\n 'value_from': None},\n {'name': 'S3_VERIFY_SSL',\n 'value': '0',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE',\n 'value': '/etc/ssl/custom-certs/cabundle.crt',\n 'value_from': None},\n {'name': 'AWS_CA_BUNDLE_CONFIGMAP',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'HF_HUB_ENABLE_HF_TRANSFER',\n 'value': '1',\n 'value_from': None},\n {'name': 'HF_XET_HIGH_PERFORMANCE',\n 'value': '1',\n 'value_fro [e2e-llm-inference-service] m': None},\n {'name': 'HF_XET_NUM_CONCURRENT_RANGE_GETS',\n 'value': '8',\n 'value_from': None},\n {'name': 'CA_BUNDLE_CONFIGMAP_NAME',\n 'value': 'odh-kserve-custom-ca-bundle',\n 'value_from': None},\n {'name': 'CA_BUNDLE_VOLUME_MOUNT_POINT',\n 'value': '/etc/ssl/custom-certs',\n 'value_from': None}],\n 'env_from': None,\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_pull_policy': 'IfNotPresent',\n 'lifecycle': None,\n 'liveness_probe': None,\n 'name': 'storage-initializer',\n 'ports': None,\n 'readiness_probe': None,\n 'resize_policy': None,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_policy': None,\n 'security_context': {'allow_privilege_escalation': False,\n 'app_armor_profile': None,\n 'capabilities': {'add': None,\n 'drop': ['ALL']},\n 'privileged': None,\n 'proc_mount': None,\n 'read_only_root_filesystem': None,\n 'run_as_group': None,\n 'run_as_non_root': True,\n 'run_as_user': 1000690000,\n 'se_linux_options': None,\n 'seccomp_profile': None,\n 'windows_options': None},\n 'startup_probe': None,\n 'stdin': None,\n 'stdin_once': None,\n 'termination_message_path': '/dev/termination-log',\n 'termination_message_policy': 'FallbackToLogsOnError',\n 'tty': None,\n 'volume_devices': None,\n 'volume_mounts': [{'mount_path': '/mnt',\n 'mount_propagation': None,\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None,\n [e2e-llm-inference-service] 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'mount_propagation': None,\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'mount_propagation': None,\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': None,\n 'sub_path': None,\n 'sub_path_expr': None}],\n 'working_dir': None}],\n 'node_name': 'ip-10-0-131-57.ec2.internal',\n 'node_selector': None,\n 'os': None,\n 'overhead': None,\n 'preemption_policy': 'PreemptLowerPriority',\n 'priority': 0,\n 'priority_class_name': None,\n 'readiness_gates': None,\n 'resource_claims': None,\n 'resources': None,\n 'restart_policy': 'Always',\n 'runtime_class_name': None,\n 'scheduler_name': 'default-scheduler',\n 'scheduling_gates': None,\n 'security_context': {'app_armor_profile': None,\n 'fs_group': 1000690000,\n 'fs_group_change_policy': None,\n 'run_as_group': None,\n 'run_as_non_root': None,\n 'run_as_user': None,\n 'se_linux_change_policy': None,\n 'se_linux_options': {'level': 's0:c26,c20',\n 'role': None,\n 'type': None,\n 'user': None},\n 'seccomp_profile': {'localhost_profile': None,\n 'type': 'RuntimeDefault'},\n 'supplemental_groups': None,\n 'supplemental_groups_policy': None,\n 'sysctls': None,\n 'windows_options': None},\n 'service_account': 'default',\n 'service_account_name': 'default',\n 'set_hostname_as_fqdn': None,\n 'share_process_namespace': None,\n 'subdomain': None,\n 'termination_grace_period_seconds': 60,\n 'tolerations': [{'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/not-ready',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n [e2e-llm-inference-service] {'effect': 'NoExecute',\n 'key': 'node.kubernetes.io/unreachable',\n 'operator': 'Exists',\n 'toleration_seconds': 300,\n 'value': None},\n {'effect': 'NoSchedule',\n 'key': 'node.kubernetes.io/memory-pressure',\n 'operator': 'Exists',\n 'toleration_seconds': None,\n 'value': None}],\n 'topology_spread_constraints': None,\n 'volumes': [{'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None,\n 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'home',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': 'Memory',\n 'size_limit': '1Gi'},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'dshm',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n [e2e-llm-inference-service] 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None,\n 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'model-cache',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None,\n 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tmp-dir',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n [e2e-llm-inference-service] 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'tls-certs',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': {'default_mode': 420,\n 'items': None,\n 'optional': None,\n 'secret_name': 'lora-multiple-adapters-test-kserve-self-signed-certs'},\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': {'medium': None,\n 'size_limit': None},\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kserve-provision-location',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': {'default_mode': 420,\n 'items': None,\n 'name': 'odh-kserve-custom-ca-bundle',\n 'optional': None},\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\ [e2e-llm-inference-service] n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'cabundle-cert',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': None,\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None},\n {'aws_elastic_block_store': None,\n 'azure_disk': None,\n 'azure_file': None,\n 'cephfs': None,\n 'cinder': None,\n 'config_map': None,\n 'csi': None,\n 'downward_api': None,\n 'empty_dir': None,\n 'ephemeral': None,\n 'fc': None,\n 'flex_volume': None,\n 'flocker': None,\n 'gce_persistent_disk': None,\n 'git_repo': None,\n 'glusterfs': None,\n 'host_path': None,\n 'image': None,\n 'iscsi': None,\n 'name': 'kube-api-access-n5dws',\n 'nfs': None,\n 'persistent_volume_claim': None,\n 'photon_persistent_disk': None,\n 'portworx_volume': None,\n 'projected': {'default_mode': 420,\n 'sources': [{'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': {'audience': None,\n 'expiration_seconds': 3607,\n 'path': 'token'}},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'ca.crt',\n 'mode': None,\n 'path': 'ca.crt'}],\n 'name': 'kube-root-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n [e2e-llm-inference-service] 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': None,\n 'downward_api': {'items': [{'field_ref': {'api_version': 'v1',\n 'field_path': 'metadata.namespace'},\n 'mode': None,\n 'path': 'namespace',\n 'resource_field_ref': None}]},\n 'secret': None,\n 'service_account_token': None},\n {'cluster_trust_bundle': None,\n 'config_map': {'items': [{'key': 'service-ca.crt',\n 'mode': None,\n 'path': 'service-ca.crt'}],\n 'name': 'openshift-service-ca.crt',\n 'optional': None},\n 'downward_api': None,\n 'secret': None,\n 'service_account_token': None}]},\n 'quobyte': None,\n 'rbd': None,\n 'scale_io': None,\n 'secret': None,\n 'storageos': None,\n 'vsphere_volume': None}]},\n 'status': {'conditions': [{'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 18, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodReadyToStartContainers'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'Initialized'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'Ready'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 36, 48, tzinfo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'ContainersReady'},\n {'last_probe_time': None,\n 'last_transition_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzin [e2e-llm-inference-service] fo=tzlocal()),\n 'message': None,\n 'reason': None,\n 'status': 'True',\n 'type': 'PodScheduled'}],\n 'container_statuses': [{'allocated_resources': {'cpu': '200m',\n 'memory': '2Gi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://d07c7d25f918ec33f58c76ee8a53505d25d03878a705942fa5818f9f2f9ec402',\n 'image': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0',\n 'image_id': 'public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'main',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '2',\n 'memory': '7Gi'},\n 'requests': {'cpu': '200m',\n 'memory': '2Gi'}},\n 'restart_count': 0,\n 'started': True,\n 'state': {'running': {'started_at': datetime.datetime(2026, 7, 6, 21, 35, 23, tzinfo=tzlocal())},\n 'terminated': None,\n 'waiting': None},\n 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/home',\n 'name': 'home',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/tmp',\n 'name': 'tmp-dir',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/dev/shm',\n 'name': 'dshm',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/models',\n 'name': 'model-cache',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/var/run/kserve/tls',\n [e2e-llm-inference-service] 'name': 'tls-certs',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'ephemeral_container_statuses': None,\n 'host_i_ps': [{'ip': '10.0.131.57'}],\n 'host_ip': '10.0.131.57',\n 'init_container_statuses': [{'allocated_resources': {'cpu': '100m',\n 'memory': '100Mi'},\n 'allocated_resources_status': None,\n 'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'image': 'quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c',\n 'image_id': 'quay.io/opendatahub/kserve-storage-initializer@sha256:60e2a5c8616d82c8933b614ae003dd3412a420a29a0be570b6a6439a3d39ab99',\n 'last_state': {'running': None,\n 'terminated': None,\n 'waiting': None},\n 'name': 'storage-initializer',\n 'ready': True,\n 'resources': {'claims': None,\n 'limits': {'cpu': '1',\n 'memory': '24Gi'},\n 'requests': {'cpu': '100m',\n 'memory': '100Mi'}},\n 'restart_count': 0,\n 'started': False,\n 'state': {'running': None,\n 'terminated': {'container_id': 'cri-o://fa571c5ccaabab45ecf0b91fa029437607a56123a0ebf6aea850e8700239be31',\n 'exit_code': 0,\n 'finished_at': datetime.datetime(2026, 7, 6, 21, 35, 22, tzinfo=tzlocal()),\n 'message': None,\n 'reason': 'Completed',\n 'signal': None,\n 'started_at': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())},\n 'waiting': None},\n [e2e-llm-inference-service] 'user': {'linux': {'gid': 0,\n 'supplemental_groups': [0,\n 1000690000],\n 'uid': 1000690000}},\n 'volume_mounts': [{'mount_path': '/mnt',\n 'name': 'kserve-provision-location',\n 'read_only': None,\n 'recursive_read_only': None},\n {'mount_path': '/etc/ssl/custom-certs',\n 'name': 'cabundle-cert',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'},\n {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',\n 'name': 'kube-api-access-n5dws',\n 'read_only': True,\n 'recursive_read_only': 'Disabled'}]}],\n 'message': None,\n 'nominated_node_name': None,\n 'phase': 'Running',\n 'pod_i_ps': [{'ip': '10.134.0.76'}],\n 'pod_ip': '10.134.0.76',\n 'qos_class': 'Burstable',\n 'reason': None,\n 'resize': None,\n 'resource_claim_statuses': None,\n 'start_time': datetime.datetime(2026, 7, 6, 21, 35, 17, tzinfo=tzlocal())}}],\n 'kind': 'PodList',\n 'metadata': {'_continue': None,\n 'remaining_item_count': None,\n 'resource_version': '79976',\n 'self_link': None}}.items [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-06T21:52:14.063317] end - ✅ in 20.456s [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_lora_adapters:test_llm_lora_adapters.py:189 Cleaning up config router-no-scheduler-lora-multip-343dff1f [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_lora_adapters:test_llm_lora_adapters.py:189 Cleaning up config workload-single-cpu-lora-multip-b14d2ed8 [e2e-llm-inference-service] INFO e2e.llmisvc.test_llm_lora_adapters:test_llm_lora_adapters.py:189 Cleaning up config model-fb-opt-125m-with-multiple-7cbd2065 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.56/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:54 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:54 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:54 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:40 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.56:8000/health": dial tcp 10.134.0.56:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:57:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:49 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-6cf96f5c75-xnsm8 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:51 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:51 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:51 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:57:05 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:57:05 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-7b76977598 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:49 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-6cf96f5c75 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:38 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:55:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:55:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:55:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:54 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:57:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:33 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:33 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:33 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:33 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:37 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:20 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 43.086s (43.086s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:20 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:20 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:03 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.49:8000/health": dial tcp 10.134.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:17 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:33 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.49:8000/health": dial tcp 10.134.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-566f9cdfc7-k88bd [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:33 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:33 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:33 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:33 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:34 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:36 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.537s (1.537s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:36 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:36 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:17 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:17 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:18 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning Unhealthy Readiness probe failed: timeout: failed to connect service "10.132.0.32:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-78df948669 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-566f9cdfc7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-enabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:51 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.53:8000/health": dial tcp 10.134.0.53:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:02 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:18 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.53:8000/health": dial tcp 10.134.0.53:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:00 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-754cb7b645-9cmz7 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:02 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:02 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:02 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:02 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:02 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-8dc8569bc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:00 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-754cb7b645 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:01 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:30 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "conv-test-criticality": llminferenceservices.serving.kserve.io "conv-test-criticality" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:30 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "conv-test-lora-crit": llminferenceservices.serving.kserve.io "conv-test-lora-crit" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:31 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "conv-test-round-trip": Operation cannot be fulfilled on llminferenceservices.serving.kserve.io "conv-test-round-trip": the object has been modified; please apply your changes to the latest version and try again [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:30 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "conv-test-v1a1-to-v1a2": llminferenceservices.serving.kserve.io "conv-test-v1a1-to-v1a2" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.74/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:20:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:20:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:20:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:23:19 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.74:8001/health": dial tcp 10.134.0.74:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-76b459f96b-75c2l [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.75/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:20:04 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:20:04 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:20:04 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:23:19 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.75:8000/health": dial tcp 10.134.0.75:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-688fff8dc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:20:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:20:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:20:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:05 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:05 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-67f64f9f55 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-76b459f96b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:52 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:20:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:20:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:20:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:23:30 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.67/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:58 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:58 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:58 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:11:33 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.67:8000/health": dial tcp 10.134.0.67:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:11:48 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.67:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:54 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:54 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:54 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:54 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:55 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:55 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:55 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:11:48 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:11:48 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-9576f666b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-7f49bcb7c9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:51 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:09:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:09:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:11:44 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:11:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/e2e-pvc-model-download-lw446 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:01:29 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:01:37 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.59/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:01:37 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:01:37 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:01:37 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:01:24 kserve-ci-e2e-test job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-lw446 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:01:47 kserve-ci-e2e-test job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:01:24 kserve-ci-e2e-test persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:01:24 kserve-ci-e2e-test persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:01:24 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-678fcd5fb4-nqx64_f90fa33d-dd79-4308-bfa8-6f23ebfe388b Normal Provisioning External provisioner is provisioning volume for claim "kserve-ci-e2e-test/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:01:26 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-678fcd5fb4-nqx64_f90fa33d-dd79-4308-bfa8-6f23ebfe388b Normal ProvisioningSucceeded Successfully provisioned volume pvc-cff970c0-1328-4734-8bb0-45cf29b3a727 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:00 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:17 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:17 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:17 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:20 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:18 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "gw-section-name-router-with-gat-f1d92d0f": Operation cannot be fulfilled on llminferenceservices.serving.kserve.io "gw-section-name-router-with-gat-f1d92d0f": the object has been modified; please apply your changes to the latest version and try again [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.79/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:45:27 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:45:27 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:45:27 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:47:14 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.79:8000/health": dial tcp 10.134.0.79:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:45:10 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-3c960099-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:45:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:47:24 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-3c960099] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.70/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:48 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:48 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:54 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:54 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:18:57 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.70:8001/health": dial tcp 10.134.0.70:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:21 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:21 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.71/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:18:57 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.71:8000/health": dial tcp 10.134.0.71:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:21 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:37 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.71:8000/health": dial tcp 10.134.0.71:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-68bd486ddb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:41 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:08 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-50bc673d] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.64/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:46 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:10 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.64:8000/health": dial tcp 10.134.0.64:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:26 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:41 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.64:8000/health": dial tcp 10.134.0.64:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:38 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:21 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:00 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:01 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:15 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:00 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:57:45 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:14 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.57/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:51 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:55 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:55 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:55 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:10 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.57:8000/health": dial tcp 10.134.0.57:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:06 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.57:8000/health": context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-64ff8447c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:51 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:51 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:51 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:51 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:52 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:52 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:52 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:50 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:50 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:46 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:57:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:57:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:57:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:21 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-7ca60146] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.58/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:36 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:36 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:36 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:00:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.58:8000/health": dial tcp 10.134.0.58:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:01:03 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:01:19 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.58:8000/health": dial tcp 10.134.0.58:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-7c5b498f8d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25 to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:33 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:33 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:33 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:01:03 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:01:03 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:31 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:59:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:01:02 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:01:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:38 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.66/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:38 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:38 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:38 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:11 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:37 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:38 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.65/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:38 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:38 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:38 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:11:18 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.65:8000/health": dial tcp 10.134.0.65:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:11:54 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:10 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.65:8000/health": dial tcp 10.134.0.65:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test leaderworkerset Normal CreatingRevision Creating revision with key 686d468674 for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:11:28 kserve-ci-e2e-test leaderworkerset Normal AllGroupsReady All replicas are ready, with 1 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:32 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-2577e794-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test37591b20e96e9663d45a730d03070f1e-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:11:50 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-pvc-router-manage-2577e794] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:11:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test37591b20e96e9663d45a730d03070f1e-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:13 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:18 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.60/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:18 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:18 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:18 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:08 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.60:8000/health": dial tcp 10.134.0.60:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:22 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:38 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.60:8000/health": dial tcp 10.134.0.60:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:04 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-59b9d263-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:18 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-pvc-router-manage-59b9d263] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:49 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:51 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.63/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 877ms (877ms including waiting). Image size: 44914394 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:07:41 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.63:8001/health": dial tcp 10.134.0.63:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:06 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:06 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:49 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:52 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.62/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:07:41 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.62:8000/health": dial tcp 10.134.0.62:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:06 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:21 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.62:8000/health": dial tcp 10.134.0.62:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:45 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:07:52 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-pvc-router-manage-e8706282] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.55/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:55:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:23 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:23 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:55:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:55:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:55:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:55:29 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4f8c0978] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:55:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:55 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:10 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.52:8000/health": dial tcp 10.134.0.52:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:45 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:41 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-a50492e9] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9 to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:06 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:43 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:17 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:12 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:32 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:06 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:29 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:29 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:29 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:29 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:02 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:01 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6fcb489785 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:55:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:55:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5 to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:36 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:36 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:37 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0, llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:11:54 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:11:54 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-54954df77 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4 to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:49 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:49 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:49 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:21 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:21 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:29 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:29 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:29 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:29 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:02 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:02 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5dd88bfbb7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.61/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:12 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:12 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:12 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:22 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:22 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-684f55d7cf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68 to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:26 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:26 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:51 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:51 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:51 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:55 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:52:55 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-6d86bd4d9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:48 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:48 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:48 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:04:49 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw, llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:06 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:08:06 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:46 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.54/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:46 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:46 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:46 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:06 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:56:06 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-68cc9685d6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-749449dbc8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:35 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:35 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:50:35 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:12 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:53:12 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.76/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:35:23 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:35:23 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:35:23 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:36:37 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.76:8000/health": dial tcp 10.134.0.76:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:51:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:52:09 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.76:8000/health": dial tcp 10.134.0.76:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: lora-multiple-adapters-test-kserve-77485bff95-2tx5t [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set lora-multiple-adapters-test-kserve-77485bff95 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:35:13 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy lora-multiple-adapters-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "lora-multiple-adapters-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/lora-multiple-adapters-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:35:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:36:48 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [lora-multiple-adapters-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:51 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.73/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:51 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:51 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:51 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:55 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:55 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:55 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:21:20 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.73:8000/health": dial tcp 10.134.0.73:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:34:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: lora-single-adapter-test-kserve-58678c44f4-sbltp [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set lora-single-adapter-test-kserve-58678c44f4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:46 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy lora-single-adapter-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/lora-single-adapter-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "lora-single-adapter-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/lora-single-adapter-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/lora-single-adapter-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/lora-single-adapter-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/lora-single-adapter-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/lora-single-adapter-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/lora-single-adapter-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:21:31 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [lora-single-adapter-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:27 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:30 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 2.786s (2.786s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:30 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:30 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:57 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:18 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.47:8000/health": dial tcp 10.134.0.47:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:49:21 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.47:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:27 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" in 4.367s (4.367s including waiting). Image size: 300633537 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:27 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:27 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:31 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:32 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 987ms (987ms including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:32 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:32 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:57 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6d789b4bf5-5whz4 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:23 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:23 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:23 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:24 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:26 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.641s (1.641s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:03 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:03 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:26 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:02 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" in 35.689s (35.689s including waiting). Image size: 2989890188 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:02 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:02 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:03 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:24 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.134.0.48:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:57 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:57 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:57 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7bc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-6d789b4bf5 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:36 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:48:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:11 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:42 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:42 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:42 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:58 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "http://10.134.0.46:15021/healthz/ready": dial tcp 10.134.0.46:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:30 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning BackOff Back-off restarting failed container istio-proxy in pod router-gateway-1-openshift-default-799f46c59b-pl8dd_kserve-ci-e2e-test(ee146378-761d-47dd-96b5-24f9cfaff899) [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:58:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.46:15021/healthz/ready": dial tcp 10.134.0.46:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-pl8dd [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-799f46c59b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:11 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 20:47:15 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:09 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:26 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:28 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 2.474s (2.474s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:28 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:28 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:34 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.52:15021/healthz/ready": dial tcp 10.132.0.52:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:25 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-54c789bdc6-22wcp [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:25 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-54c789bdc6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:25 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:29 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:25 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-2 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.77/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:51 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:44:25 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.77:8001/health": dial tcp 10.134.0.77:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:44:44 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:44:44 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.78/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:49 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:44:25 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.78:8000/health": dial tcp 10.134.0.78:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:44:44 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-prefill-7754f68459 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8 to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:46 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:46 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:46 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:44:44 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:44:44 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-6f7778f59 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-65ddc8b8c8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:41:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:44:35 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:44:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.68/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:17 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:17 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:17 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:20 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:20 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:20 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:14:36 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.68:8000/health": dial tcp 10.134.0.68:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:14:54 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:09 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.68:8000/health": dial tcp 10.134.0.68:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-5cf96c9bd7-bbfd9 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:17 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:17 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:17 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:18 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:18 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:18 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:14:54 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:14:54 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-router-scheduler-7dc958f8d6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-5cf96c9bd7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:14:47 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:14:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.69/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:39 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:39 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:39 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:14:49 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.69:8000/health": dial tcp 10.134.0.69:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:16 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.69:8000/health": dial tcp 10.134.0.69:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.72/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:14 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:14 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:14 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:10 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.72:8000/health": dial tcp 10.134.0.72:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:23 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:38 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.72:8000/health": dial tcp 10.134.0.72:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: stop-feature-test-kserve-9b9d66cf-2xd45 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: stop-feature-test-kserve-9b9d66cf-9snw6 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:12 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:12 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:12 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:23 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:23 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:39 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:39 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:39 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:40 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:40 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:40 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set stop-feature-test-kserve-router-scheduler-85c9fdd856 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set stop-feature-test-kserve-router-scheduler-85c9fdd856 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set stop-feature-test-kserve-9b9d66cf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:16:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set stop-feature-test-kserve-9b9d66cf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:41 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy stop-feature-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "stop-feature-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-stop-feature-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/stop-feature-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/stop-feature-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/stop-feature-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:12:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:19:20 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [stop-feature-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.InferencePool kserve-ci-e2e-test/stop-feature-test-inference-pool [e2e-llm-inference-service] INFO e2e.llmisvc.diagnostic:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test LLMInferenceServiceController Warning LLMInferenceServiceNotReady LLMInferenceService [stop-feature-test] is no longer Ready because of: GatewaysReady, HTTPRoutesReady, InferencePoolReady, MainWorkloadReady, PrefillWorkerWorkloadReady, PrefillWorkloadReady, RouterReady, SchedulerWorkloadReady, WorkerWorkloadReady, WorkloadsReady [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_with_lora_adapters] [2026-07-06T21:52:14.710436] end - ❌ 1026.198s: HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/lora-multiple-adapters-test/v1/completions (Caused by ReadTimeoutError("HTTPConnectionPool(host='a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")) [e2e-llm-inference-service] _ test_llm_inference_service[router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'scheduler-v06-pd-config-migration', 'workload-llmd-simulator-pd'], prompt='KSer... {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] print(f"{prefix} Waiting for model response from {service_name}") [e2e-llm-inference-service] > wait_for_model_response( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] test_case, [e2e-llm-inference-service] test_case.wait_timeout, [e2e-llm-inference-service] extra_headers=test_case.extra_headers, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:816: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, TestCase(base_refs=['router-managed', 'scheduler-v0... {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900) [e2e-llm-inference-service] kwargs = {'extra_headers': None}, func_name = 'wait_for_model_response' [e2e-llm-inference-service] timestamp_start = '2026-07-06T22:04:22.329920', start_time = 1783375462.33029 [e2e-llm-inference-service] duration = 1103.057543516159, timestamp_end = '2026-07-06T22:22:45.387835' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'scheduler-v06-pd-config-migration', 'workload-llmd-simulator-pd'], prompt='KSer... {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] timeout_seconds = 900, extra_headers = None [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_model_response( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] test_case: TestCase, # noqa: F811 [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] extra_headers: Optional[Dict[str, str]] = None, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] > response = wait_for(get_successful_response, timeout=timeout_seconds, interval=5.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1119: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .get_successful_response at 0x7fbb6911c4a0> [e2e-llm-inference-service] timeout = 900, interval = 5.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1215: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Service returned 502: [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1115: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig router-managed-scheduler-v06-pd-5a5debb2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig router-managed-scheduler-v06-pd-5a5debb2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig router-managed-scheduler-v06-pd-5a5debb2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig scheduler-v06-pd-config-migrati-bb3264a2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig scheduler-v06-pd-config-migrati-bb3264a2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig scheduler-v06-pd-config-migrati-bb3264a2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig workload-llmd-simulator-pd-sche-64d0fc8b in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-pd-sche-64d0fc8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-pd-sche-64d0fc8b [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-06T22:03:42.439873] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'scheduler-v06-pd-config-migration', 'workload-llmd-simulator-pd'], prompt='KServe is a', service_name='scheduler-v06-pd-migration-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-06T22:03:42.454092] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-06T22:03:42.523910] end - ✅ in 0.070s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-06T22:03:42.524151] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-06T22:03:51Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T22:03:51Z', 'message': 'Inference Pool kserve-ci-e2e-test/scheduler-v06-pd-migration-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-06T22:03:51Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-06T22:03:51Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-06T22:03:51Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T22:03:51Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T22:03:51Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T22:03:51Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T22:03:51Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-06T22:04:12Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: "False" (reason "BackendNotFound", message "backend(scheduler-v06-pd-migration-test-inference-pool-ip-2bbd07cf.kserve-ci-e2e-test.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T22:04:12Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-06T22:04:12Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-06T22:04:12Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-06T22:03:51Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T22:04:12Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: "False" (reason "BackendNotFound", message "backend(scheduler-v06-pd-migration-test-inference-pool-ip-2bbd07cf.kserve-ci-e2e-test.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T22:04:12Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: "False" (reason "BackendNotFound", message "backend(scheduler-v06-pd-migration-test-inference-pool-ip-2bbd07cf.kserve-ci-e2e-test.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T22:04:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T22:04:12Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-06T22:04:18Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T22:04:12Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-06T22:04:12Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-06T22:04:12Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-06T22:03:51Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T22:04:18Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T22:04:18Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T22:04:12Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T22:04:12Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_ready] [2026-07-06T22:04:22.329610] end - ✅ in 39.805s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_model_response] [2026-07-06T22:04:22.329920] start - args=(, TestCase(base_refs=['router-managed', 'scheduler-v06-pd-config-migration', 'workload-llmd-simulator-pd'], prompt='KServe is a', service_name='scheduler-v06-pd-migration-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900), kwargs={'extra_headers': None} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-06T22:04:22.330293] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-06T22:04:22.338071] end - ✅ in 0.007s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 502: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Service returned 502: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-06T22:10:31.684739] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-06T22:10:31.738525] end - ✅ in 0.053s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 502: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-06T22:16:41.029491] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-06T22:16:41.087727] end - ✅ in 0.058s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 502: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1219 Timed out waiting: Service returned 502: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_model_response] [2026-07-06T22:22:45.387835] end - ❌ 1103.058s: Service returned 502: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:831 [router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] ❌ ERROR: Failed to call llm inference service scheduler-v06-pd-migration-test: Service returned 502: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1240 🔍 # Diagnostics for 'scheduler-v06-pd-migration-test' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1241 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1242 # LLMInferenceService scheduler-v06-pd-migration-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1245 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:42Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T22:03:42Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T22:03:42Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:prefill: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T22:04:21Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '91114' [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-scheduler-v06-pd-5a5debb2 [e2e-llm-inference-service] - name: scheduler-v06-pd-config-migrati-bb3264a2 [e2e-llm-inference-service] - name: workload-llmd-simulator-pd-sche-64d0fc8b [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/scheduler-v06-pd-migration-test [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/scheduler-v06-pd-migration-test [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T22:04:18Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T22:04:12Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T22:04:12Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T22:04:12Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PrefillWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T22:03:51Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T22:04:21Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T22:04:21Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T22:04:21Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T22:04:12Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a6c719f352bff4273af2e2b406e6cfaa-1772363841.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/scheduler-v06-pd-migration-test [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-6cf96f5c75-xnsm8 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.56/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:54 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:54 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:54 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:40 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.56:8000/health": dial tcp 10.134.0.56:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:57:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:49 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-6cf96f5c75-xnsm8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:51 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:51 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:51 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:57:05 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:57:05 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-7b76977598-jng7l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-7b76977598 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:49 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-6cf96f5c75 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:38 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:55:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:55:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:55:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:54 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:57:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-566f9cdfc7-k88bd to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:33 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:33 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:33 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:33 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:37 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:20 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 43.086s (43.086s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:20 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:20 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:03 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.49:8000/health": dial tcp 10.134.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:17 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:33 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.49:8000/health": dial tcp 10.134.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-566f9cdfc7-k88bd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:33 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:33 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:33 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:33 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:34 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:36 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.537s (1.537s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:36 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:36 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:17 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:17 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:18 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning Unhealthy Readiness probe failed: timeout: failed to connect service "10.132.0.32:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-78df948669-xnfzn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-78df948669 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-566f9cdfc7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:13 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-enabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-754cb7b645-9cmz7 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:51 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.53:8000/health": dial tcp 10.134.0.53:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:02 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:18 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.53:8000/health": dial tcp 10.134.0.53:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:00 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-754cb7b645-9cmz7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:02 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:02 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:02 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:02 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:02 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-8dc8569bc-9lfkp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-8dc8569bc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:00 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-754cb7b645 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:01 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:30 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "conv-test-criticality": llminferenceservices.serving.kserve.io "conv-test-criticality" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:30 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "conv-test-lora-crit": llminferenceservices.serving.kserve.io "conv-test-lora-crit" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:31 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "conv-test-round-trip": Operation cannot be fulfilled on llminferenceservices.serving.kserve.io "conv-test-round-trip": the object has been modified; please apply your changes to the latest version and try again [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:30 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "conv-test-v1a1-to-v1a2": llminferenceservices.serving.kserve.io "conv-test-v1a1-to-v1a2" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-76b459f96b-75c2l to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.74/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:19 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.74:8001/health": dial tcp 10.134.0.74:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-76b459f96b-75c2l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.75/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:04 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:04 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:04 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:19 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.75:8000/health": dial tcp 10.134.0.75:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-prefill-688fff8dc-c5k8l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-688fff8dc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-67f64v46sd to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:05 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:05 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-67f64f9f55 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-76b459f96b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:52 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:30 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.67/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:58 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:58 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:58 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:33 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.67:8000/health": dial tcp 10.134.0.67:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:48 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:05 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.67:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-7f49bcb7c9-xrz4d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-9576f666g4dnr to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:54 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:54 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:54 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:54 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:55 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:55 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:55 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:48 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:48 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-9576f666b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-7f49bcb7c9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:51 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:44 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/e2e-pvc-model-download-lw446 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:01:29 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:01:37 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.59/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:01:37 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:01:37 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:01:37 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:01:24 kserve-ci-e2e-test job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-lw446 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:01:47 kserve-ci-e2e-test job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:01:24 kserve-ci-e2e-test persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:01:24 kserve-ci-e2e-test persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:01:24 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-678fcd5fb4-nqx64_f90fa33d-dd79-4308-bfa8-6f23ebfe388b Normal Provisioning External provisioner is provisioning volume for claim "kserve-ci-e2e-test/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:01:26 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-678fcd5fb4-nqx64_f90fa33d-dd79-4308-bfa8-6f23ebfe388b Normal ProvisioningSucceeded Successfully provisioned volume pvc-cff970c0-1328-4734-8bb0-45cf29b3a727 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc54ffqd to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:00 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4r6kn to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:17 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:17 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:17 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:20 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:18 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "gw-section-name-router-with-gat-f1d92d0f": Operation cannot be fulfilled on llminferenceservices.serving.kserve.io "gw-section-name-router-with-gat-f1d92d0f": the object has been modified; please apply your changes to the latest version and try again [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556k4n5s to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.79/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:45:27 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:45:27 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:45:27 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:47:14 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.79:8000/health": dial tcp 10.134.0.79:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:53:29 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:53:44 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.79:8000/health": dial tcp 10.134.0.79:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-3c960099-kserve-79d6779556 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:45:10 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-3c960099-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:45:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:45:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:47:24 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-3c960099] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.70/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:48 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:48 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:54 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:54 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:57 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.70:8001/health": dial tcp 10.134.0.70:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:21 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:21 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7-62tgp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.71/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:57 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.71:8000/health": dial tcp 10.134.0.71:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:21 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:37 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.71:8000/health": dial tcp 10.134.0.71:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-689p8q4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-68bd486ddb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-76f8ff7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:41 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:08 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-50bc673d] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bfk9jnt to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.64/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:46 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:10 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.64:8000/health": dial tcp 10.134.0.64:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:26 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:41 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.64:8000/health": dial tcp 10.134.0.64:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c75767bf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:38 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:21 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.82/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:07 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:07 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:11 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:11 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:11 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:58:29 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.81/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:07 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:07 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:12 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:12 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:12 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:56:26 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.81:8000/health": dial tcp 10.134.0.81:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:58:12 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:58:27 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.81:8000/health": dial tcp 10.134.0.81:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test leaderworkerset Normal CreatingRevision Creating revision with key 6884774d76 for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:56:36 kserve-ci-e2e-test leaderworkerset Normal AllGroupsReady All replicas are ready, with 1 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:53:59 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test386bd5808c5c450f11fd8632e1f821fb-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:58:07 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-dc21cb14] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:58:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test386bd5808c5c450f11fd8632e1f821fb-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:00 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4jnhnv to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:01 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:15 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:00 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:57:45 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:14 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.57/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:51 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:55 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:55 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:55 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:10 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.57:8000/health": dial tcp 10.134.0.57:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:06 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.57:8000/health": context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd-hgbbf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-64ff8447c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schew9b6t to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:51 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:51 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:51 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:51 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:52 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:52 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:52 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:50 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:50 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74bcfd5dd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:46 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:57:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:57:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:57:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:21 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-7ca60146] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.58/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:36 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:36 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:36 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:00:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.58:8000/health": dial tcp 10.134.0.58:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:01:03 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:01:19 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.58:8000/health": dial tcp 10.134.0.58:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c-wkg9j [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-7c5b498f8d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheptl25 to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:33 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:33 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:33 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:01:03 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:01:03 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5b785bb88c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:31 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:59:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:01:02 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:01:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:38 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.66/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:38 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:38 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:38 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:11 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:37 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:38 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.65/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:38 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:38 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:38 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:18 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.65:8000/health": dial tcp 10.134.0.65:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:54 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:10 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.65:8000/health": dial tcp 10.134.0.65:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test leaderworkerset Normal CreatingRevision Creating revision with key 686d468674 for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:28 kserve-ci-e2e-test leaderworkerset Normal AllGroupsReady All replicas are ready, with 1 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:32 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-2577e794-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test37591b20e96e9663d45a730d03070f1e-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:50 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-pvc-router-manage-2577e794] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test37591b20e96e9663d45a730d03070f1e-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5nfq7 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:13 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:18 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.60/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:18 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:18 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:18 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:08 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.60:8000/health": dial tcp 10.134.0.60:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:22 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:38 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.60:8000/health": dial tcp 10.134.0.60:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:04 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-59b9d263-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:18 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-pvc-router-manage-59b9d263] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:49 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:51 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.63/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 877ms (877ms including waiting). Image size: 44914394 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:41 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.63:8001/health": dial tcp 10.134.0.63:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:06 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:06 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:49 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:52 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.62/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:41 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.62:8000/health": dial tcp 10.134.0.62:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:06 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:21 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.62:8000/health": dial tcp 10.134.0.62:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:45 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:52 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-pvc-router-manage-e8706282] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff47f89g to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.87/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:23 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:23 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:23 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:57 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:23 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:16 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test80f99418f0f3eb4c80f02e6144a6d4f2-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:56 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4c1ba212] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test80f99418f0f3eb4c80f02e6144a6d4f2-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fwcrjd to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.55/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:55:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:23 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:23 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:55:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:55:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:55:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:55:29 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4f8c0978] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:55:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddfjv7d to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:55 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:10 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.52:8000/health": dial tcp 10.134.0.52:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:45 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:41 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-a50492e9] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9 to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:06 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-r6wj9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:43 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:17 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4jm6mj to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:12 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:32 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:06 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879fh85d to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:29 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:29 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:29 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:29 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:02 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:01 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6fcb489785 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegqg2s to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:54:24 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:55:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:55:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schep27c5 to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:35 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:36 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:36 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:37 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0, llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:54 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:54 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.55/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:07 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:07 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:07 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:08 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:08 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:08 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:58:12 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:58:12 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebnpsz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler-76b968d679 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:23 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler-79866dc455 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schekbjfp to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.88/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:24 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:24 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:24 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:57 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:57 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-54954df77 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schelwzf4 to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:47 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:49 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:49 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:49 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:21 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:21 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:29 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:29 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:29 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:29 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:02 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:02 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec67qk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:28 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5dd88bfbb7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.61/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:12 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:12 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:12 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:22 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:22 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheb7fwg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:40 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-684f55d7cf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schem2p68 to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:26 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:26 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:51 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:51 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:51 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:55 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:52:55 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4r85b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:51:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-6d86bd4d9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schestptt to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:48 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:48 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:48 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:49 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-cff970c0-1328-4734-8bb0-45cf29b3a727" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79xgqrw, llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mn5kv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:06 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:06 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:46 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.54/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:46 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:46 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:46 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:06 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:56:06 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche92nln [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-68cc9685d6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-749449dbc8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schek9fzc to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:34 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:35 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:35 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:50:35 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:12 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:53:12 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77485bff95-2tx5t to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.76/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:23 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:23 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:23 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:36:37 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.76:8000/health": dial tcp 10.134.0.76:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:51:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:09 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.76:8000/health": dial tcp 10.134.0.76:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: lora-multiple-adapters-test-kserve-77485bff95-2tx5t [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set lora-multiple-adapters-test-kserve-77485bff95 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:13 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy lora-multiple-adapters-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "lora-multiple-adapters-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/lora-multiple-adapters-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:36:48 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [lora-multiple-adapters-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/lora-single-adapter-test-kserve-58678c44f4-sbltp to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:51 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.73/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:51 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:51 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:51 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:55 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:55 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:55 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:20 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.73:8000/health": dial tcp 10.134.0.73:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:34:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: lora-single-adapter-test-kserve-58678c44f4-sbltp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set lora-single-adapter-test-kserve-58678c44f4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:46 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy lora-single-adapter-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/lora-single-adapter-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "lora-single-adapter-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/lora-single-adapter-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/lora-single-adapter-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/lora-single-adapter-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/lora-single-adapter-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/lora-single-adapter-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/lora-single-adapter-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:31 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [lora-single-adapter-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-5whz4 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:27 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:30 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 2.786s (2.786s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:30 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:30 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:57 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:18 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.47:8000/health": dial tcp 10.134.0.47:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:49:21 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.47:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:27 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" in 4.367s (4.367s including waiting). Image size: 300633537 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:27 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:27 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:31 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:32 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 987ms (987ms including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:32 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:32 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:57 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6d789b4bf5-5whz4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6d789b4bf5-k4mhv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7wqxr9 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:23 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:23 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:23 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:24 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:26 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.641s (1.641s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:03 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:03 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:26 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:02 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" in 35.689s (35.689s including waiting). Image size: 2989890188 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:02 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:02 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:03 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:24 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.134.0.48:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:57 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:57 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:57 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-75c4fbc7bc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-6d789b4bf5 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:36 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:48:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-ckl2g to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.83/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:46 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:46 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:57:35 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.83:8000/health": dial tcp 10.134.0.83:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:57:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/prestop-hook-test-kserve-56bd76dbb7-sgkcg to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:57:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.84/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:57:48 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:57:48 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:57:48 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:57:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:57:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:57:53 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:57:58 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.84:8000/health": dial tcp 10.134.0.84:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:58:06 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: prestop-hook-test-kserve-56bd76dbb7-ckl2g [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:57:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: prestop-hook-test-kserve-56bd76dbb7-sgkcg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:46 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.56/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:46 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:46 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:46 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:47 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:47 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:47 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:58:06 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning Unhealthy Liveness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:58:06 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:58:06 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: prestop-hook-test-kserve-router-scheduler-6ff4cbb856-858dr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set prestop-hook-test-kserve-router-scheduler-6ff4cbb856 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set prestop-hook-test-kserve-56bd76dbb7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/prestop-hook-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/prestop-hook-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/prestop-hook-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/prestop-hook-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-prestop-hook-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/prestop-hook-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/prestop-hook-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:55:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/prestop-hook-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:56:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/prestop-hook-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:57:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/prestop-hook-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:56:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/prestop-hook-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:57:46 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [prestop-hook-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:57:52 kserve-ci-e2e-test LLMInferenceServiceController Warning LLMInferenceServiceNotReady LLMInferenceService [prestop-hook-test] is no longer Ready because of: MainWorkloadReady, WorkloadsReady [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:58:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-prestop-hook-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-wf42c to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.86/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:19 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:19 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:19 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:02 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:18 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: rolling-upgrade-test-kserve-54bc76f55b-wf42c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: rolling-upgrade-test-kserve-54bc76f55b-wf42c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-xcph8 to ip-10-0-142-77.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:52 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.26/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:52 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:53 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.007s (1.007s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:53 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:53 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:25 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:41 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.26:8000/health": dial tcp 10.133.0.26:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:51 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: rolling-upgrade-test-kserve-859df57bcb-xcph8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.58/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:19 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:19 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:19 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:25 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:26 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:18 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-9j95r [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:18 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:18 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set rolling-upgrade-test-kserve-54bc76f55b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:51 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set rolling-upgrade-test-kserve-859df57bcb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set rolling-upgrade-test-kserve-54bc76f55b from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:17 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy rolling-upgrade-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/rolling-upgrade-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "rolling-upgrade-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/rolling-upgrade-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/rolling-upgrade-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/rolling-upgrade-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-rolling-upgrade-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/rolling-upgrade-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/rolling-upgrade-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/rolling-upgrade-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/rolling-upgrade-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/rolling-upgrade-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:50 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [rolling-upgrade-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Deployment kserve-ci-e2e-test/rolling-upgrade-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:00:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-rolling-upgrade-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pl8dd to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:11 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:42 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:42 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:42 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:58 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "http://10.134.0.46:15021/healthz/ready": dial tcp 10.134.0.46:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:30 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning BackOff Back-off restarting failed container istio-proxy in pod router-gateway-1-openshift-default-799f46c59b-pl8dd_kserve-ci-e2e-test(ee146378-761d-47dd-96b5-24f9cfaff899) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:58:47 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.46:15021/healthz/ready": dial tcp 10.134.0.46:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-pl8dd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-799f46c59b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:11 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 20:47:15 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:09 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-22wcp to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:26 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:28 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 2.474s (2.474s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:28 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:28 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:34 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.52:15021/healthz/ready": dial tcp 10.132.0.52:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:25 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-54c789bdc6-22wcp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:25 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-54c789bdc6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:25 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:29 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:25 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.77/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:51 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:52 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:44:25 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.77:8001/health": dial tcp 10.134.0.77:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:44:44 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:44:44 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-65ddc8b8c8-g9t47 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.78/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:49 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:44:25 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.78:8000/health": dial tcp 10.134.0.78:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:44:44 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-prefill-7754f68459-hpqd5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-prefill-7754f68459 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-6f7778f59rlbw8 to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:46 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:46 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:46 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:44:44 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:44:44 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-6f7778f59 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-65ddc8b8c8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:41:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:44:35 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:44:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-5cf96c9bd7-bbfd9 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.68/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:17 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:17 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:17 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:20 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:20 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:20 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:36 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.68:8000/health": dial tcp 10.134.0.68:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:54 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:09 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.68:8000/health": dial tcp 10.134.0.68:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-5cf96c9bd7-bbfd9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:17 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:17 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:17 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:18 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:18 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:18 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:54 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:54 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-router-scheduler-7dc958f8d6-98s2g [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-router-scheduler-7dc958f8d6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-5cf96c9bd7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:47 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.60/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:19 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:19 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:19 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:10 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-configmap-ref-test-kserve-744bf59fbd-zxb4h [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:20 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.89/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:20 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:20 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:20 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:10 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:10 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-configmap-ref-test-kserve-router-scheduler-68f874qbtb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-configmap-ref-test-kserve-router-scheduler-68f876cc5d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-configmap-ref-test-kserve-744bf59fbd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:18 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-configmap-ref-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-configmap-ref-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:01:38 kserve-ci-e2e-test LLMInferenceServiceController Warning Error Reconciliation failed: failed to reconcile workload: failed to reconcile self-signed certificates secret: failed to reconcile self-signed TLS certificate: failed to get defaults for v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs: context canceled [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:08 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-configmap-ref-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-vz68r to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.90/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:36 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:36 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:36 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:22 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-5bc7bd6864-vz68r [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-router-scheduler-567wx8kj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567wx8kj to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.91/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:36 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:36 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:36 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:22 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:22 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-router-scheduler-567469f665 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-5bc7bd6864 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:32 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-custom-template-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-custom-template-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-custom-template-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-custom-template-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-custom-template-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-custom-template-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:21 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-custom-template-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.61/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:23 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:49 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-5c5cfcf776-8gmqg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dl5nhm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85djfn6t to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.92/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:23 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:23 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dl5nhm to ip-10-0-142-77.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.27/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:52 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.697s (1.697s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:52 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:52 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:23 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:23 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:23 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbc4 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:49 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-5c5cfcf776 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:40 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-ha-replicas-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-ha-replicas-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:02:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:22 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-ha-replicas-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:01 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.57/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:01 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:34 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:01 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-inline-config-test-kserve-7c6c58cc96-p2x9t [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:01 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsdl4t to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.85/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:02 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:02 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:02 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:34 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:34 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:01 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-inline-config-test-kserve-router-scheduler-d9ddf6c4d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:01 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-inline-config-test-kserve-7c6c58cc96 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:58:58 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-inline-config-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-inline-config-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-inline-config-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-inline-config-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-inline-config-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-inline-config-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-inline-config-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-inline-config-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:33 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-inline-config-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:59:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:49 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.93/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:49 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:49 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:49 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:50 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:49 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:49 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.62/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:49 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:49 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:49 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:49 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:48 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:49 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.94/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:49 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:49 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:49 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:49 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:49 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-router-scheduler-59dc679cd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:48 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-5c87756974 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:47 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-v06-pd-migration-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-v06-pd-migration-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-v06-pd-migration-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:04:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:04:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:04:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:04:21 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-v06-pd-migration-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-f4slq to ip-10-0-142-77.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:56 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:57 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 680ms (680ms including waiting). Image size: 44914394 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:57 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:57 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:58 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:58 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:58 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:04:29 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:04:29 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:04:45 kserve-ci-e2e-test kubelet/ip-10-0-142-77.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.28:8001/health": dial tcp 10.133.0.28:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-threshold-migration-test-kserve-5994569d5b-f4slq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zpgnx to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.63/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:56 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:56 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:56 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:04:29 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-threshold-migration-test-kserve-prefill-5fc599dc49 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.95/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:56 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:56 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:56 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:04:29 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:04:29 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-threshold-migration-test-kserve-router-sched6w6zm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-threshold-migration-test-kserve-router-scheduler-dbc65f645 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-threshold-migration-test-kserve-5994569d5b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:54 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-v06-threshold-migration-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-v06-threshold-migration-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:03:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test0a07e82be2036793543b2d3db2ea0785-scheduler-v0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:04:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:04:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:04:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:04:28 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-v06-threshold-migration-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 22:04:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test0a07e82be2036793543b2d3db2ea0785-scheduler-v0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-2xd45 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.69/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:39 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:39 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:39 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:45 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:49 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.69:8000/health": dial tcp 10.134.0.69:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:16 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.69:8000/health": dial tcp 10.134.0.69:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-9b9d66cf-9snw6 to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.72/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:14 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:14 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:14 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:10 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.72:8000/health": dial tcp 10.134.0.72:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:23 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:38 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.72:8000/health": dial tcp 10.134.0.72:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: stop-feature-test-kserve-9b9d66cf-2xd45 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: stop-feature-test-kserve-9b9d66cf-9snw6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:12 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:12 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:12 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:23 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:23 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:39 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:39 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:39 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:40 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:40 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:40 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: stop-feature-test-kserve-router-scheduler-85c9fdd856-jg8ml [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: stop-feature-test-kserve-router-scheduler-85c9fdd856-8sv2s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set stop-feature-test-kserve-router-scheduler-85c9fdd856 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:10 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set stop-feature-test-kserve-router-scheduler-85c9fdd856 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set stop-feature-test-kserve-9b9d66cf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set stop-feature-test-kserve-9b9d66cf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:41 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy stop-feature-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "stop-feature-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-stop-feature-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/stop-feature-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/stop-feature-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/stop-feature-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:20 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [stop-feature-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.InferencePool kserve-ci-e2e-test/stop-feature-test-inference-pool [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:00 kserve-ci-e2e-test LLMInferenceServiceController Warning LLMInferenceServiceNotReady LLMInferenceService [stop-feature-test] is no longer Ready because of: GatewaysReady, HTTPRoutesReady, InferencePoolReady, MainWorkloadReady, PrefillWorkerWorkloadReady, PrefillWorkloadReady, RouterReady, SchedulerWorkloadReady, WorkerWorkloadReady, WorkloadsReady [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/tls-verification-test-kserve-85684c76c-44jqz to ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.80/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:36 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:36 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:36 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:40 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:41 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:41 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:26 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.80:8000/health": dial tcp 10.134.0.80:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:41 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:58 kserve-ci-e2e-test kubelet/ip-10-0-131-57.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.80:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: tls-verification-test-kserve-85684c76c-44jqz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb to ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:37 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.54/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:37 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:9b7f9014cc3727e4fd95a945572d3f70017252d596bd8941a1887e4a1514875c" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:37 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:37 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:38 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:38 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:38 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:41 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:41 kserve-ci-e2e-test kubelet/ip-10-0-131-45.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: tls-verification-test-kserve-router-scheduler-68d855bcfb-plwbb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set tls-verification-test-kserve-router-scheduler-68d855bcfb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set tls-verification-test-kserve-85684c76c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:35 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy tls-verification-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/tls-verification-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "tls-verification-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/tls-verification-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/tls-verification-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-tls-verification-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/tls-verification-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/tls-verification-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/tls-verification-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:53:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/tls-verification-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:53:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:53:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/tls-verification-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:37 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [tls-verification-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:54:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-tls-verification-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'llm-d-routing-sidecar' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 Flag --decoder-use-tls has been deprecated, use --enable-tls=decoder instead [e2e-llm-inference-service] Flag --prefiller-use-tls has been deprecated, use --enable-tls=prefiller instead [e2e-llm-inference-service] {"level":"info","ts":1783375429.5569925,"msg":"Proxy starting","Built on":"v0.9.0-rc.2","From Git SHA":"181aa8358916e19b8844ccc752b2d6153d4b2ad6"} [e2e-llm-inference-service] {"level":"info","ts":1783375429.5570233,"msg":"Proxy configuration","config":"{\"Port\":\"8000\",\"KVConnector\":\"nixlv2\",\"ECConnector\":\"\",\"DataParallelSize\":1,\"MaxIdleConnsPerHost\":1024,\"EnablePrefillerSampling\":false,\"PrefillMaxRetries\":0,\"PrefillRetryBackoff\":200000000,\"UseTLSForPrefiller\":true,\"UseTLSForDecoder\":true,\"UseTLSForEncoder\":false,\"InsecureSkipVerifyForPrefiller\":false,\"InsecureSkipVerifyForEncoder\":false,\"InsecureSkipVerifyForDecoder\":false,\"SecureServing\":true,\"CertPath\":\"/var/run/kserve/tls\",\"MooncakeBootstrapPort\":8998,\"EnableSSRFProtection\":true,\"InferencePoolNamespace\":\"kserve-ci-e2e-test\",\"InferencePoolName\":\"scheduler-v06-pd-migration-test-inference-pool\",\"PoolGroup\":\"inference.networking.x-k8s.io\",\"DecodeChunkSize\":0,\"Tracing\":false,\"MoRIIOWriteMode\":false,\"MoRIIODecodeNotifyPort\":61005,\"MoRIIODecodeHandshakePort\":6301,\"MoRIIODecodePodIP\":\"\",\"MoRIIOParallelDispatch\":false,\"MoRIIOPrefillHandshakePort\":6301,\"MoRIIOPrefillNotifyPort\":61005,\"MoRIIOTPSize\":1,\"MoRIIODPSize\":1,\"MoRIIORemoteHosts\":null,\"MoRIIODPSizeLocal\":0,\"MoRIIODecodeHosts\":null,\"DecoderURL\":\"https://localhost:8001\"}"} [e2e-llm-inference-service] {"level":"info","ts":1783375429.5576198,"logger":"allowlist-validator","msg":"starting SSRF protection allowlist validator","namespace":"kserve-ci-e2e-test","poolName":"scheduler-v06-pd-migration-test-inference-pool","gvr":"inference.networking.x-k8s.io/v1alpha2, Resource=inferencepools"} [e2e-llm-inference-service] {"level":"info","ts":1783375429.5707097,"logger":"allowlist-validator","msg":"InferencePool added","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375429.5780246,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":0,"targets":{}} [e2e-llm-inference-service] {"level":"info","ts":1783375429.578107,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":0,"targets":{}} [e2e-llm-inference-service] {"level":"info","ts":1783375429.6579223,"logger":"allowlist-validator","msg":"allowlist validator started successfully"} [e2e-llm-inference-service] {"level":"info","ts":1783375429.6597893,"logger":"proxy server on port 8000","msg":"server TLS configured"} [e2e-llm-inference-service] {"level":"info","ts":1783375429.6598039,"logger":"proxy server on port 8000","msg":"starting","addr":"[::]:8000"} [e2e-llm-inference-service] {"level":"info","ts":1783375429.7408755,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.132.0.62":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375430.2470062,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375431.2264113,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375439.3643727,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375439.3773723,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375439.382794,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375439.7932265,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375440.2939956,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375459.571438,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375459.579474,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375459.5795338,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] 2026/07/06 22:04:22 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783375462.4090664,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"a4e0ca08-7986-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/06 22:04:22 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783375462.4274704,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"a4e3d441-7986-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/06 22:04:26 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783375466.4462702,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"a749007c-7986-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/06 22:04:34 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783375474.4665916,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"ac10c2e5-7986-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783375489.5724354,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375489.5813606,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375489.58142,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] 2026/07/06 22:04:50 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783375490.512835,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"b5a12839-7986-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783375519.5734327,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375519.5810554,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375519.581112,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] 2026/07/06 22:05:22 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783375522.5342817,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"c8b737f1-7986-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783375549.5744789,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375549.583126,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375549.5831788,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375579.574702,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375579.5831282,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375579.5831847,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] 2026/07/06 22:06:26 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783375586.5830956,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"eee4342c-7986-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783375609.5755262,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375609.5835087,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375609.583558,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375639.576242,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375639.5909185,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375639.590993,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375669.5771077,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375669.5860298,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375669.5860927,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375699.5775366,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375699.5855896,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375699.5856514,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] 2026/07/06 22:08:26 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783375706.6337001,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"3672892e-7987-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783375729.5783072,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375729.5868473,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375729.5869,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375759.5793786,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375759.5874586,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375759.5875127,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375789.5804472,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375789.5896926,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375789.5897446,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375819.5814288,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375819.5897198,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375819.589777,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] 2026/07/06 22:10:26 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783375826.6815317,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"7e007056-7987-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/06 22:10:31 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783375831.784743,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"810b47f4-7987-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/06 22:10:31 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783375831.8030958,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"810e1466-7987-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/06 22:10:35 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783375835.820878,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"83731f81-7987-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/06 22:10:43 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783375843.8389232,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"883a9686-7987-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783375849.5824225,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375849.5919511,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375849.5920558,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] 2026/07/06 22:10:59 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783375859.8608732,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"91c71afb-7987-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783375879.5834105,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375879.5950577,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375879.5951176,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] 2026/07/06 22:11:31 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783375891.882434,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"a4dd4177-7987-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783375909.584287,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375909.5933182,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375909.5933738,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375939.5844674,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375939.5939744,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375939.5940495,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] 2026/07/06 22:12:35 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783375955.9301353,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"cb0a2835-7987-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783375969.5854506,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375969.5942025,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375969.5942533,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375999.586492,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783375999.5955067,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783375999.5955591,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376029.5873828,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783376029.5973034,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376029.5973694,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376059.5878513,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783376059.597711,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376059.5977652,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] 2026/07/06 22:14:35 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783376075.975637,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"1297a963-7988-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783376089.5884035,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783376089.5981493,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376089.5982094,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376119.5894227,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783376119.5970106,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376119.597059,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376149.5904517,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783376149.5993085,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376149.5993738,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376179.59138,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783376179.6007378,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376179.6007922,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] 2026/07/06 22:16:36 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783376196.0265484,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"5a25ff69-7988-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/06 22:16:41 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783376201.129948,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"5d3099c3-7988-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/06 22:16:41 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783376201.1490245,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"5d33790e-7988-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/06 22:16:45 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783376205.167627,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"5f98f8bd-7988-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783376209.592364,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783376209.6015654,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376209.6016233,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] 2026/07/06 22:16:53 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783376213.1862013,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"64608543-7988-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/06 22:17:09 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783376229.2088857,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"6ded331b-7988-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783376239.5924165,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783376239.6005619,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376239.6006186,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] 2026/07/06 22:17:41 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783376261.231849,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"81038670-7988-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783376269.5928156,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783376269.6023877,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376269.6024463,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376299.593521,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783376299.6015203,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376299.6015751,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] 2026/07/06 22:18:45 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783376325.286029,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"a7315a65-7988-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783376329.5939963,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783376329.60226,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376329.6023138,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376359.5940335,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783376359.6025898,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376359.6026452,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376389.594305,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783376389.6028314,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376389.6028848,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376419.594539,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783376419.603259,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376419.6033165,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] 2026/07/06 22:20:45 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783376445.3323376,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"eebf0ee1-7988-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783376449.5954072,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783376449.6038227,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376449.604065,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376479.5961044,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783376479.6050315,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376479.6050844,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376509.5964327,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783376509.6047406,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376509.6047966,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376539.5972111,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783376539.605138,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783376539.6051958,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.62":{},"10.134.0.93":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh":{}}} [e2e-llm-inference-service] 2026/07/06 22:22:45 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.132.0.62 [e2e-llm-inference-service] {"level":"error","ts":1783376565.3852243,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"364dae23-7989-11f1-b5ca-0a580a86005d","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 I0706 22:03:50.277065 1 config.go:602] "Configuration:" =< [e2e-llm-inference-service] { [e2e-llm-inference-service] "IP": "", [e2e-llm-inference-service] "PodName": "", [e2e-llm-inference-service] "PodNameSpace": "", [e2e-llm-inference-service] "VllmDevMode": false, [e2e-llm-inference-service] "block-size": 16, [e2e-llm-inference-service] "data-parallel-rank": -1, [e2e-llm-inference-service] "data-parallel-size": 1, [e2e-llm-inference-service] "dataset-in-memory": false, [e2e-llm-inference-service] "dataset-path": "", [e2e-llm-inference-service] "dataset-table-name": "llmd", [e2e-llm-inference-service] "dataset-url": "", [e2e-llm-inference-service] "default-embedding-dimensions": 384, [e2e-llm-inference-service] "ec-transfer-config": "", [e2e-llm-inference-service] "enable-kvcache": false, [e2e-llm-inference-service] "enable-prefix-caching": false, [e2e-llm-inference-service] "enable-request-id-headers": false, [e2e-llm-inference-service] "enable-sleep-mode": false, [e2e-llm-inference-service] "enforce-eager": false, [e2e-llm-inference-service] "event-batch-size": 16, [e2e-llm-inference-service] "failure-injection-rate": 0, [e2e-llm-inference-service] "failure-types": null, [e2e-llm-inference-service] "fake-metrics": null, [e2e-llm-inference-service] "fake-metrics-refresh-interval": 100000000, [e2e-llm-inference-service] "global-cache-hit-threshold": 0, [e2e-llm-inference-service] "hash-seed": "", [e2e-llm-inference-service] "inter-token-latency": 0, [e2e-llm-inference-service] "inter-token-latency-std-dev": 0, [e2e-llm-inference-service] "kv-cache-size": 1024, [e2e-llm-inference-service] "kv-cache-transfer-latency": 0, [e2e-llm-inference-service] "kv-cache-transfer-latency-std-dev": 0, [e2e-llm-inference-service] "kv-cache-transfer-time-per-token": 0, [e2e-llm-inference-service] "kv-cache-transfer-time-std-dev": 0, [e2e-llm-inference-service] "latency-calculator": "", [e2e-llm-inference-service] "lora-modules": null, [e2e-llm-inference-service] "max-cpu-loras": 1, [e2e-llm-inference-service] "max-loras": 1, [e2e-llm-inference-service] "max-model-len": 1024, [e2e-llm-inference-service] "max-num-seqs": 5, [e2e-llm-inference-service] "max-tool-call-array-param-length": 5, [e2e-llm-inference-service] "max-tool-call-integer-param": 100, [e2e-llm-inference-service] "max-tool-call-number-param": 100, [e2e-llm-inference-service] "max-waiting-queue-length": 1000, [e2e-llm-inference-service] "min-tool-call-array-param-length": 1, [e2e-llm-inference-service] "min-tool-call-integer-param": 0, [e2e-llm-inference-service] "min-tool-call-number-param": 0, [e2e-llm-inference-service] "mm-encoder-only": false, [e2e-llm-inference-service] "mm-processor-kwargs": "", [e2e-llm-inference-service] "mode": "random", [e2e-llm-inference-service] "model": "facebook/opt-125m", [e2e-llm-inference-service] "object-tool-call-not-required-field-probability": 50, [e2e-llm-inference-service] "port": 8001, [e2e-llm-inference-service] "prefill-overhead": 0, [e2e-llm-inference-service] "prefill-time-per-token": 0, [e2e-llm-inference-service] "prefill-time-std-dev": 0, [e2e-llm-inference-service] "seed": 1783375430276605200, [e2e-llm-inference-service] "self-signed-certs": false, [e2e-llm-inference-service] "served-model-name": [ [e2e-llm-inference-service] "facebook/opt-125m" [e2e-llm-inference-service] ], [e2e-llm-inference-service] "ssl-certfile": "/var/run/kserve/tls/tls.crt", [e2e-llm-inference-service] "ssl-keyfile": "/var/run/kserve/tls/tls.key", [e2e-llm-inference-service] "time-factor-under-load": 1, [e2e-llm-inference-service] "time-to-first-token": 0, [e2e-llm-inference-service] "time-to-first-token-std-dev": 0, [e2e-llm-inference-service] "tool-call-not-required-param-probability": 50, [e2e-llm-inference-service] "uds-socket-path": "/tmp/tokenizer/tokenizer-uds.socket", [e2e-llm-inference-service] "zmq-endpoint": "tcp://127.0.0.1:5557" [e2e-llm-inference-service] } [e2e-llm-inference-service] > [e2e-llm-inference-service] I0706 22:03:50.310849 1 tokenizer.go:104] "Model is not a real HF model, using simulated tokenizer" model="facebook/opt-125m" [e2e-llm-inference-service] I0706 22:03:50.316166 1 context.go:138] "No dataset path or URL provided, using random text for responses" [e2e-llm-inference-service] I0706 22:03:50.316233 1 simulator.go:188] "Start processing routine" [e2e-llm-inference-service] I0706 22:03:50.316273 1 communication.go:49] "Starting communication layer" [e2e-llm-inference-service] I0706 22:03:50.316564 1 http_server_tls.go:44] "HTTPS server starting with certificate files" cert="/var/run/kserve/tls/tls.crt" key="/var/run/kserve/tls/tls.key" [e2e-llm-inference-service] I0706 22:03:50.316612 1 grpc.go:126] "Server starting" protocol="gRPC" port=8001 [e2e-llm-inference-service] I0706 22:03:50.317363 1 http.go:96] "Server starting" protocol="HTTPS" port=8001 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 I0706 22:03:49.581283 1 config.go:602] "Configuration:" =< [e2e-llm-inference-service] { [e2e-llm-inference-service] "IP": "", [e2e-llm-inference-service] "PodName": "", [e2e-llm-inference-service] "PodNameSpace": "", [e2e-llm-inference-service] "VllmDevMode": false, [e2e-llm-inference-service] "block-size": 16, [e2e-llm-inference-service] "data-parallel-rank": -1, [e2e-llm-inference-service] "data-parallel-size": 1, [e2e-llm-inference-service] "dataset-in-memory": false, [e2e-llm-inference-service] "dataset-path": "", [e2e-llm-inference-service] "dataset-table-name": "llmd", [e2e-llm-inference-service] "dataset-url": "", [e2e-llm-inference-service] "default-embedding-dimensions": 384, [e2e-llm-inference-service] "ec-transfer-config": "", [e2e-llm-inference-service] "enable-kvcache": false, [e2e-llm-inference-service] "enable-prefix-caching": false, [e2e-llm-inference-service] "enable-request-id-headers": false, [e2e-llm-inference-service] "enable-sleep-mode": false, [e2e-llm-inference-service] "enforce-eager": false, [e2e-llm-inference-service] "event-batch-size": 16, [e2e-llm-inference-service] "failure-injection-rate": 0, [e2e-llm-inference-service] "failure-types": null, [e2e-llm-inference-service] "fake-metrics": null, [e2e-llm-inference-service] "fake-metrics-refresh-interval": 100000000, [e2e-llm-inference-service] "global-cache-hit-threshold": 0, [e2e-llm-inference-service] "hash-seed": "", [e2e-llm-inference-service] "inter-token-latency": 0, [e2e-llm-inference-service] "inter-token-latency-std-dev": 0, [e2e-llm-inference-service] "kv-cache-size": 1024, [e2e-llm-inference-service] "kv-cache-transfer-latency": 0, [e2e-llm-inference-service] "kv-cache-transfer-latency-std-dev": 0, [e2e-llm-inference-service] "kv-cache-transfer-time-per-token": 0, [e2e-llm-inference-service] "kv-cache-transfer-time-std-dev": 0, [e2e-llm-inference-service] "latency-calculator": "", [e2e-llm-inference-service] "lora-modules": null, [e2e-llm-inference-service] "max-cpu-loras": 1, [e2e-llm-inference-service] "max-loras": 1, [e2e-llm-inference-service] "max-model-len": 1024, [e2e-llm-inference-service] "max-num-seqs": 5, [e2e-llm-inference-service] "max-tool-call-array-param-length": 5, [e2e-llm-inference-service] "max-tool-call-integer-param": 100, [e2e-llm-inference-service] "max-tool-call-number-param": 100, [e2e-llm-inference-service] "max-waiting-queue-length": 1000, [e2e-llm-inference-service] "min-tool-call-array-param-length": 1, [e2e-llm-inference-service] "min-tool-call-integer-param": 0, [e2e-llm-inference-service] "min-tool-call-number-param": 0, [e2e-llm-inference-service] "mm-encoder-only": false, [e2e-llm-inference-service] "mm-processor-kwargs": "", [e2e-llm-inference-service] "mode": "random", [e2e-llm-inference-service] "model": "facebook/opt-125m", [e2e-llm-inference-service] "object-tool-call-not-required-field-probability": 50, [e2e-llm-inference-service] "port": 8000, [e2e-llm-inference-service] "prefill-overhead": 0, [e2e-llm-inference-service] "prefill-time-per-token": 0, [e2e-llm-inference-service] "prefill-time-std-dev": 0, [e2e-llm-inference-service] "seed": 1783375429580880000, [e2e-llm-inference-service] "self-signed-certs": false, [e2e-llm-inference-service] "served-model-name": [ [e2e-llm-inference-service] "facebook/opt-125m" [e2e-llm-inference-service] ], [e2e-llm-inference-service] "ssl-certfile": "/var/run/kserve/tls/tls.crt", [e2e-llm-inference-service] "ssl-keyfile": "/var/run/kserve/tls/tls.key", [e2e-llm-inference-service] "time-factor-under-load": 1, [e2e-llm-inference-service] "time-to-first-token": 0, [e2e-llm-inference-service] "time-to-first-token-std-dev": 0, [e2e-llm-inference-service] "tool-call-not-required-param-probability": 50, [e2e-llm-inference-service] "uds-socket-path": "/tmp/tokenizer/tokenizer-uds.socket", [e2e-llm-inference-service] "zmq-endpoint": "tcp://127.0.0.1:5557" [e2e-llm-inference-service] } [e2e-llm-inference-service] > [e2e-llm-inference-service] I0706 22:03:49.618714 1 tokenizer.go:104] "Model is not a real HF model, using simulated tokenizer" model="facebook/opt-125m" [e2e-llm-inference-service] I0706 22:03:49.624415 1 context.go:138] "No dataset path or URL provided, using random text for responses" [e2e-llm-inference-service] I0706 22:03:49.624634 1 communication.go:49] "Starting communication layer" [e2e-llm-inference-service] I0706 22:03:49.624719 1 simulator.go:188] "Start processing routine" [e2e-llm-inference-service] I0706 22:03:49.625013 1 http_server_tls.go:44] "HTTPS server starting with certificate files" cert="/var/run/kserve/tls/tls.crt" key="/var/run/kserve/tls/tls.key" [e2e-llm-inference-service] I0706 22:03:49.625041 1 grpc.go:126] "Server starting" protocol="gRPC" port=8000 [e2e-llm-inference-service] I0706 22:03:49.625706 1 http.go:96] "Server starting" protocol="HTTPS" port=8000 [e2e-llm-inference-service] I0706 22:04:22.408990 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:41136\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:04:22.427491 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:41144\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:04:26.446268 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:41154\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:04:34.466584 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:59804\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:04:50.512860 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:55172\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:05:22.534256 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:54108\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:06:26.583015 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:41304\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:08:26.633688 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:59114\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:10:26.681477 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:39266\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:10:31.784683 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:39956\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:10:31.803085 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:39972\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:10:35.820843 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:39974\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:10:43.838895 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:54546\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:10:59.860846 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:40748\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:11:31.882389 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:34542\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:12:35.930103 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:59460\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:14:35.975620 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:39534\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:16:36.026562 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:33120\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:16:41.130033 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:33122\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:16:41.149035 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:33128\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:16:45.167615 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:44748\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:16:53.186212 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:36602\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:17:09.208896 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:60870\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:17:41.231870 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:40354\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:18:45.285995 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:33404\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:20:45.332350 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:57398\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0706 22:22:45.385180 1 communication.go:105] "Server error" msg="error when serving connection \"10.132.0.62:8000\"<->\"10.134.0.93:39204\": remote error: tls: bad certificate" [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 "Type": "STRING", [e2e-llm-inference-service] "Value": "181aa8358916e19b8844ccc752b2d6153d4b2ad6" [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] ] [e2e-llm-inference-service] }, [e2e-llm-inference-service] "InstrumentationLibrary": { [e2e-llm-inference-service] "Name": "llm-d-router/pkg/epp/framework/plugins/scheduling", [e2e-llm-inference-service] "Version": "v0.9.0-rc.2", [e2e-llm-inference-service] "SchemaURL": "", [e2e-llm-inference-service] "Attributes": [ [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "commit-sha", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "STRING", [e2e-llm-inference-service] "Value": "181aa8358916e19b8844ccc752b2d6153d4b2ad6" [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] ] [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] { [e2e-llm-inference-service] "Name": "gateway.request_orchestration", [e2e-llm-inference-service] "SpanContext": { [e2e-llm-inference-service] "TraceID": "59e79aa8925c94341313494c055ca720", [e2e-llm-inference-service] "SpanID": "b2af33d83766068e", [e2e-llm-inference-service] "TraceFlags": "01", [e2e-llm-inference-service] "TraceState": "", [e2e-llm-inference-service] "Remote": false [e2e-llm-inference-service] }, [e2e-llm-inference-service] "Parent": { [e2e-llm-inference-service] "TraceID": "59e79aa8925c94341313494c055ca720", [e2e-llm-inference-service] "SpanID": "cab133cec9d4e45d", [e2e-llm-inference-service] "TraceFlags": "01", [e2e-llm-inference-service] "TraceState": "", [e2e-llm-inference-service] "Remote": false [e2e-llm-inference-service] }, [e2e-llm-inference-service] "SpanKind": 2, [e2e-llm-inference-service] "StartTime": "2026-07-06T22:16:41.133114296Z", [e2e-llm-inference-service] "EndTime": "2026-07-06T22:16:41.133258973Z", [e2e-llm-inference-service] "Attributes": [ [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "target_model", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "STRING", [e2e-llm-inference-service] "Value": "facebook/opt-125m" [e2e-llm-inference-service] } [e2e-llm-inference-service] }, [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "request_prio", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "INT64", [e2e-llm-inference-service] "Value": 0 [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] ], [e2e-llm-inference-service] "Events": null, [e2e-llm-inference-service] "Links": null, [e2e-llm-inference-service] "Status": { [e2e-llm-inference-service] "Code": "Unset", [e2e-llm-inference-service] "Description": "" [e2e-llm-inference-service] }, [e2e-llm-inference-service] "DroppedAttributes": 0, [e2e-llm-inference-service] "DroppedEvents": 0, [e2e-llm-inference-service] "DroppedLinks": 0, [e2e-llm-inference-service] "ChildSpanCount": 5, [e2e-llm-inference-service] "Resource": [ [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "service.name", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "STRING", [e2e-llm-inference-service] "Value": "llm-d-epp" [e2e-llm-inference-service] } [e2e-llm-inference-service] }, [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "service.version", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "STRING", [e2e-llm-inference-service] "Value": "v0.9.0-rc.2" [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] ], [e2e-llm-inference-service] "InstrumentationScope": { [e2e-llm-inference-service] "Name": "llm-d-router/pkg/epp/requestcontrol", [e2e-llm-inference-service] "Version": "v0.9.0-rc.2", [e2e-llm-inference-service] "SchemaURL": "", [e2e-llm-inference-service] "Attributes": [ [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "commit-sha", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "STRING", [e2e-llm-inference-service] "Value": "181aa8358916e19b8844ccc752b2d6153d4b2ad6" [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] ] [e2e-llm-inference-service] }, [e2e-llm-inference-service] "InstrumentationLibrary": { [e2e-llm-inference-service] "Name": "llm-d-router/pkg/epp/requestcontrol", [e2e-llm-inference-service] "Version": "v0.9.0-rc.2", [e2e-llm-inference-service] "SchemaURL": "", [e2e-llm-inference-service] "Attributes": [ [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "commit-sha", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "STRING", [e2e-llm-inference-service] "Value": "181aa8358916e19b8844ccc752b2d6153d4b2ad6" [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] ] [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] { [e2e-llm-inference-service] "Name": "gateway.request", [e2e-llm-inference-service] "SpanContext": { [e2e-llm-inference-service] "TraceID": "59e79aa8925c94341313494c055ca720", [e2e-llm-inference-service] "SpanID": "cab133cec9d4e45d", [e2e-llm-inference-service] "TraceFlags": "01", [e2e-llm-inference-service] "TraceState": "", [e2e-llm-inference-service] "Remote": false [e2e-llm-inference-service] }, [e2e-llm-inference-service] "Parent": { [e2e-llm-inference-service] "TraceID": "00000000000000000000000000000000", [e2e-llm-inference-service] "SpanID": "0000000000000000", [e2e-llm-inference-service] "TraceFlags": "00", [e2e-llm-inference-service] "TraceState": "", [e2e-llm-inference-service] "Remote": false [e2e-llm-inference-service] }, [e2e-llm-inference-service] "SpanKind": 2, [e2e-llm-inference-service] "StartTime": "2026-07-06T22:16:41.133070052Z", [e2e-llm-inference-service] "EndTime": "2026-07-06T22:16:41.149753208Z", [e2e-llm-inference-service] "Attributes": null, [e2e-llm-inference-service] "Events": null, [e2e-llm-inference-service] "Links": null, [e2e-llm-inference-service] "Status": { [e2e-llm-inference-service] "Code": "Unset", [e2e-llm-inference-service] "Description": "" [e2e-llm-inference-service] }, [e2e-llm-inference-service] "DroppedAttributes": 0, [e2e-llm-inference-service] "DroppedEvents": 0, [e2e-llm-inference-service] "DroppedLinks": 0, [e2e-llm-inference-service] "ChildSpanCount": 1, [e2e-llm-inference-service] "Resource": [ [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "service.name", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "STRING", [e2e-llm-inference-service] "Value": "llm-d-epp" [e2e-llm-inference-service] } [e2e-llm-inference-service] }, [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "service.version", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "STRING", [e2e-llm-inference-service] "Value": "v0.9.0-rc.2" [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] ], [e2e-llm-inference-service] "InstrumentationScope": { [e2e-llm-inference-service] "Name": "llm-d-router/pkg/epp/handlers", [e2e-llm-inference-service] "Version": "v0.9.0-rc.2", [e2e-llm-inference-service] "SchemaURL": "", [e2e-llm-inference-service] "Attributes": [ [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "commit-sha", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "STRING", [e2e-llm-inference-service] "Value": "181aa8358916e19b8844ccc752b2d6153d4b2ad6" [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] ] [e2e-llm-inference-service] }, [e2e-llm-inference-service] "InstrumentationLibrary": { [e2e-llm-inference-service] "Name": "llm-d-router/pkg/epp/handlers", [e2e-llm-inference-service] "Version": "v0.9.0-rc.2", [e2e-llm-inference-service] "SchemaURL": "", [e2e-llm-inference-service] "Attributes": [ [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "commit-sha", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "STRING", [e2e-llm-inference-service] "Value": "181aa8358916e19b8844ccc752b2d6153d4b2ad6" [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] ] [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] {"level":"info","ts":1783376205.1531336,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"fb25aaa7-b0ef-48b5-a76c-955934d0c8e4"} [e2e-llm-inference-service] {"level":"info","ts":1783376205.1534283,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"fb25aaa7-b0ef-48b5-a76c-955934d0c8e4","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783376213.1716495,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"b216c6c6-521a-4788-a827-e3fb4d0d231d"} [e2e-llm-inference-service] {"level":"info","ts":1783376213.1719546,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"b216c6c6-521a-4788-a827-e3fb4d0d231d","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783376229.1930497,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"61a7a7ed-179a-4f2b-a352-3c0675e03b38"} [e2e-llm-inference-service] {"level":"info","ts":1783376229.1933696,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"61a7a7ed-179a-4f2b-a352-3c0675e03b38","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783376261.2162974,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"dc3381de-2bbb-486e-9377-d4c6fd3d8714"} [e2e-llm-inference-service] {"level":"info","ts":1783376261.2165847,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"dc3381de-2bbb-486e-9377-d4c6fd3d8714","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783376325.269047,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"b941dfdb-2e7d-433c-9933-3b75549635cc"} [e2e-llm-inference-service] {"level":"info","ts":1783376325.2694263,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"b941dfdb-2e7d-433c-9933-3b75549635cc","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783376445.3154242,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"48d00786-40a3-41af-87db-fb25110ecfa0"} [e2e-llm-inference-service] {"level":"info","ts":1783376445.3157785,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"48d00786-40a3-41af-87db-fb25110ecfa0","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783376565.36791,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"dd1a71ed-00ec-493c-bc72-90960b43f65a"} [e2e-llm-inference-service] {"level":"info","ts":1783376565.3682442,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"dd1a71ed-00ec-493c-bc72-90960b43f65a","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 855d039d-6757-48e4-b8ad-7163090730b4 [e2e-llm-inference-service] resourceVersion: '91100' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-06T22:04:21Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:04:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.94 [e2e-llm-inference-service] nodeName: ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb [e2e-llm-inference-service] uid: 112b1f94-cac9-40bf-aa75-74c27b4e1c45 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 820650b7-ec5e-4ec0-b2d7-9fa911ec301d [e2e-llm-inference-service] resourceVersion: '90825' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-06T22:04:00Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:04:00Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.62 [e2e-llm-inference-service] nodeName: ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh [e2e-llm-inference-service] uid: ad17c5df-6533-4378-b417-259efe93ba3c [e2e-llm-inference-service] - ip: 10.134.0.93 [e2e-llm-inference-service] nodeName: ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm [e2e-llm-inference-service] uid: b5607a6f-b210-4689-adce-f822443fecc2 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-kserve-5c87756974- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: b5607a6f-b210-4689-adce-f822443fecc2 [e2e-llm-inference-service] resourceVersion: '90824' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c87756974 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.93/23"],"mac_address":"0a:58:0a:86:00:5d","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.93/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.93\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:5d\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c87756974 [e2e-llm-inference-service] uid: d770abd0-b36f-4bdc-90b0-6eb6afbcc662 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d770abd0-b36f-4bdc-90b0-6eb6afbcc662"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: ip-10-0-131-57 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:04:00Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.93"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-htfgq [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=kserve-ci-e2e-test/scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-htfgq [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8001' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-htfgq [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] nodeName: ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-6qcsh [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-kserve-dockercfg-xsff2 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:03:50Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:03:50Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:04:00Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:04:00Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] hostIP: 10.0.131.57 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.131.57 [e2e-llm-inference-service] podIP: 10.134.0.93 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.93 [e2e-llm-inference-service] startTime: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-router-disagg-sidecar@sha256:1ec2c035216ea0a4f8a6f7ccda55398fc3d9f71f487c135f49ba0dea08163f12 [e2e-llm-inference-service] containerID: cri-o://6923bc7e6cce08069327ab7df09d02c1d827f93623c3fa2f2ab11f28d27c2136 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-htfgq [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-06T22:03:50Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-sim@sha256:bab162bd25e2ed8b15022387cdb223023aeb33be49476af9f0115c0398fb8ff5 [e2e-llm-inference-service] containerID: cri-o://4b0be2e7a466f5d57f0d08321dde9cc77a46d6a768f5139c5b55c7e7a0155f39 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-htfgq [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: ad17c5df-6533-4378-b417-259efe93ba3c [e2e-llm-inference-service] resourceVersion: '90808' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 78b7f4c847 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.62/23"],"mac_address":"0a:58:0a:84:00:3e","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.62/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.62\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:3e\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847 [e2e-llm-inference-service] uid: c88262c6-c9f0-413b-a933-a6f554c62090 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c88262c6-c9f0-413b-a933-a6f554c62090"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: ip-10-0-131-45 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.62"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-vc6lv [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-vc6lv [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-6qcsh [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:03:59Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:03:59Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] hostIP: 10.0.131.45 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.131.45 [e2e-llm-inference-service] podIP: 10.132.0.62 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.62 [e2e-llm-inference-service] startTime: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-sim@sha256:bab162bd25e2ed8b15022387cdb223023aeb33be49476af9f0115c0398fb8ff5 [e2e-llm-inference-service] containerID: cri-o://74be01163ec95fc2b85536f6be3d0a1ce8269b7960270e545e7b07cc09be017f [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-vc6lv [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-kserve-router-scheduler-59dc679cd- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 112b1f94-cac9-40bf-aa75-74c27b4e1c45 [e2e-llm-inference-service] resourceVersion: '91099' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 59dc679cd [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.94/23"],"mac_address":"0a:58:0a:86:00:5e","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.94/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.94\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:5e\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-59dc679cd [e2e-llm-inference-service] uid: a9f550e5-c052-4e31-82c2-88a97a165516 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-131-57 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a9f550e5-c052-4e31-82c2-88a97a165516"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:04:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.94"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-58bs4 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill:\ [e2e-llm-inference-service] \ always-disagg-pd-decider\n type: disagg-profile-handler\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: max-score-picker\nschedulingProfiles:\n- name:\ [e2e-llm-inference-service] \ prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: queue-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef:\ [e2e-llm-inference-service] \ max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-58bs4 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-6qcsh [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-epp-sa-dockercfg-fdf5k [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:03:50Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:04:21Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:04:21Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] hostIP: 10.0.131.57 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.131.57 [e2e-llm-inference-service] podIP: 10.134.0.94 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.94 [e2e-llm-inference-service] startTime: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-router-endpoint-picker@sha256:06b6c75d77afd0e07053402752a9736c2dfbc12a306d0d37d963aac4c1d4e6a6 [e2e-llm-inference-service] containerID: cri-o://2356ea177209361c1b5440acf299b9086af563ad94428c20ef8038708633f04d [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-58bs4 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 1139993b-5f85-4c5d-bbe4-4eb80ed166e2 [e2e-llm-inference-service] resourceVersion: '90330' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: scheduler-v06-pd-migration-test-epp-sa-dockercfg-fdf5k [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"scheduler-v06-pd-migration-test-epp-sa-dockercfg-fdf5k"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-6qcsh"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-6qcsh [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-epp-sa-dockercfg-fdf5k [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-6qcsh [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-epp-sa-dockercfg-fdf5k [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 391fdbce-8944-47f4-9e76-56f651331e8f [e2e-llm-inference-service] resourceVersion: '90292' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: scheduler-v06-pd-migration-test-kserve-dockercfg-xsff2 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"scheduler-v06-pd-migration-test-kserve-dockercfg-xsff2"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-6qcsh"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-6qcsh [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-kserve-dockercfg-xsff2 [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-6qcsh [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-kserve-dockercfg-xsff2 [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: ce68985e-e58f-4fa3-9bf2-aa645bbf664c [e2e-llm-inference-service] resourceVersion: '90358' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.144.20 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.144.20 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0253e18a-0671-42c3-9916-833a9538a7cd [e2e-llm-inference-service] resourceVersion: '90311' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.253.54 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.253.54 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 7a93153a-d54c-4c47-8ecb-9be4a006a32b [e2e-llm-inference-service] resourceVersion: '90828' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T22:04:00Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=kserve-ci-e2e-test/scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8001' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-06T22:04:00Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:04:00Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-06T22:04:00Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-pd-migration-test-kserve-5c87756974" has successfully [e2e-llm-inference-service] progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 179505fa-01fd-41bd-81d5-4608a08e41a1 [e2e-llm-inference-service] resourceVersion: '90813' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-06T22:03:59Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:03:59Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-06T22:03:59Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 2d188ace-8c4b-47c2-986a-594fab9cbb69 [e2e-llm-inference-service] resourceVersion: '91103' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T22:04:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type:\ [e2e-llm-inference-service] \ decode-filter\n- type: always-disagg-pd-decider\n- parameters:\n deciders:\n\ [e2e-llm-inference-service] \ prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: queue-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-06T22:04:21Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:04:21Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-06T22:04:21Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-59dc679cd" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c87756974 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d770abd0-b36f-4bdc-90b0-6eb6afbcc662 [e2e-llm-inference-service] resourceVersion: '90827' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c87756974 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] uid: 7a93153a-d54c-4c47-8ecb-9be4a006a32b [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7a93153a-d54c-4c47-8ecb-9be4a006a32b"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T22:04:00Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c87756974 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c87756974 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=kserve-ci-e2e-test/scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8001' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: c88262c6-c9f0-413b-a933-a6f554c62090 [e2e-llm-inference-service] resourceVersion: '90811' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 78b7f4c847 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill [e2e-llm-inference-service] uid: 179505fa-01fd-41bd-81d5-4608a08e41a1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"179505fa-01fd-41bd-81d5-4608a08e41a1"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 78b7f4c847 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 78b7f4c847 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-59dc679cd [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: a9f550e5-c052-4e31-82c2-88a97a165516 [e2e-llm-inference-service] resourceVersion: '91102' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 59dc679cd [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler [e2e-llm-inference-service] uid: 2d188ace-8c4b-47c2-986a-594fab9cbb69 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2d188ace-8c4b-47c2-986a-594fab9cbb69"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T22:04:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 59dc679cd [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 59dc679cd [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type:\ [e2e-llm-inference-service] \ decode-filter\n- type: always-disagg-pd-decider\n- parameters:\n deciders:\n\ [e2e-llm-inference-service] \ prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: queue-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 1cda9d2d-4d17-4fe0-9fd6-944492b1986e [e2e-llm-inference-service] resourceVersion: '90352' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 2af40367-f7eb-4a3c-96e0-2da844239d56 [e2e-llm-inference-service] resourceVersion: '90299' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: cb1c82b4-8d3a-40e2-af3b-91587ee84ff3 [e2e-llm-inference-service] resourceVersion: '90348' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 939b9cf2-5f08-47f8-962f-a14d7f5634ad [e2e-llm-inference-service] resourceVersion: '90296' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service-g76sm [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 21f0730b-d604-4798-a3eb-cff95b259508 [e2e-llm-inference-service] resourceVersion: '91101' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-06T22:04:21Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] uid: ce68985e-e58f-4fa3-9bf2-aa645bbf664c [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T22:04:21Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"ce68985e-e58f-4fa3-9bf2-aa645bbf664c"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.94 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb [e2e-llm-inference-service] uid: 112b1f94-cac9-40bf-aa75-74c27b4e1c45 [e2e-llm-inference-service] nodeName: ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc-rqrjg [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 208d4e97-d60d-4f31-9a01-8abe997d96b2 [e2e-llm-inference-service] resourceVersion: '90826' [e2e-llm-inference-service] generation: 5 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-06T22:04:00Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] uid: 0253e18a-0671-42c3-9916-833a9538a7cd [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T22:04:00Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"0253e18a-0671-42c3-9916-833a9538a7cd"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.62 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh [e2e-llm-inference-service] uid: ad17c5df-6533-4378-b417-259efe93ba3c [e2e-llm-inference-service] nodeName: ip-10-0-131-45.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.93 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm [e2e-llm-inference-service] uid: b5607a6f-b210-4689-adce-f822443fecc2 [e2e-llm-inference-service] nodeName: ip-10-0-131-57.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 1cda9d2d-4d17-4fe0-9fd6-944492b1986e [e2e-llm-inference-service] resourceVersion: '90352' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 2af40367-f7eb-4a3c-96e0-2da844239d56 [e2e-llm-inference-service] resourceVersion: '90299' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: cb1c82b4-8d3a-40e2-af3b-91587ee84ff3 [e2e-llm-inference-service] resourceVersion: '90348' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 939b9cf2-5f08-47f8-962f-a14d7f5634ad [e2e-llm-inference-service] resourceVersion: '90296' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T22:03:48Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T22:04:10Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T22:04:10Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T22:04:11Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] resourceVersion: '90972' [e2e-llm-inference-service] uid: 04d7634f-e066-46ff-8119-163c952f528e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T22:04:10Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T22:04:10Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T22:04:10Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T22:04:11Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] resourceVersion: '90972' [e2e-llm-inference-service] uid: 04d7634f-e066-46ff-8119-163c952f528e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T22:04:10Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T22:04:10Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] resourceVersion: '90953' [e2e-llm-inference-service] uid: 0ba2925b-1e21-4892-802c-0d8fd6ed039c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T22:04:10Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T22:04:10Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:51Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T22:03:51Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T22:03:52Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] resourceVersion: '90538' [e2e-llm-inference-service] uid: 4d09c735-0637-4ec8-a620-27238c2cbc46 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T22:03:52Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T22:03:52Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] resourceVersion: '90389' [e2e-llm-inference-service] uid: b8039187-9c90-4611-b678-96ab93bb518b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:04:10Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T22:04:10Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] resourceVersion: '90958' [e2e-llm-inference-service] uid: 37d1766a-0ca4-4489-b8f7-aaaf8d3c6397 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-inference-pool-ip-2bbd07cf.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] resourceVersion: '90415' [e2e-llm-inference-service] uid: dc7eb778-3732-43ed-b91b-dcadaeb3f53c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] resourceVersion: '90389' [e2e-llm-inference-service] uid: b8039187-9c90-4611-b678-96ab93bb518b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:04:10Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T22:04:10Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] resourceVersion: '90958' [e2e-llm-inference-service] uid: 37d1766a-0ca4-4489-b8f7-aaaf8d3c6397 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-inference-pool-ip-2bbd07cf.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] resourceVersion: '90415' [e2e-llm-inference-service] uid: dc7eb778-3732-43ed-b91b-dcadaeb3f53c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] resourceVersion: '90389' [e2e-llm-inference-service] uid: b8039187-9c90-4611-b678-96ab93bb518b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:04:10Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T22:04:10Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] resourceVersion: '90958' [e2e-llm-inference-service] uid: 37d1766a-0ca4-4489-b8f7-aaaf8d3c6397 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-inference-pool-ip-2bbd07cf.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] resourceVersion: '90415' [e2e-llm-inference-service] uid: dc7eb778-3732-43ed-b91b-dcadaeb3f53c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"dfeab731-0d03-4689-8b34-ccfb2531cdd3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T22:03:49Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: dfeab731-0d03-4689-8b34-ccfb2531cdd3 [e2e-llm-inference-service] resourceVersion: '90366' [e2e-llm-inference-service] uid: 521f481a-0034-4188-88f9-435cb1279d6e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:22:47Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c87756974 [e2e-llm-inference-service] timestamp: '2026-07-06T22:22:20Z' [e2e-llm-inference-service] window: 22.586s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 13119277n [e2e-llm-inference-service] memory: 19408Ki [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 19879368n [e2e-llm-inference-service] memory: 24800Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:22:47Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 78b7f4c847 [e2e-llm-inference-service] timestamp: '2026-07-06T22:22:34Z' [e2e-llm-inference-service] window: 21.596s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 17397110n [e2e-llm-inference-service] memory: 25160Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-06T22:22:47Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 59dc679cd [e2e-llm-inference-service] timestamp: '2026-07-06T22:22:21Z' [e2e-llm-inference-service] window: 18.275s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 19775923n [e2e-llm-inference-service] memory: 28352Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-06T22:22:47.979110] end - ❌ 1145.539s: Service returned 502: [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:93: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] ---------- generated xml file: /workspace/artifacts-dir/junit_e2e.xml ---------- [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] ====== 2 failed, 40 passed, 3 skipped, 29 warnings in 5739.49s (1:35:39) ======= [must-gather] [must-gather ] OUT 2026-07-06T22:22:49.010573227Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 45957e43-6e07-4af2-b514-0fb39f986eca [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.23" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-06T22:22:49.060950729Z namespace/openshift-must-gather-b5t69 created [must-gather] [must-gather ] OUT 2026-07-06T22:22:49.069102468Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-srgbh created [must-gather] [must-gather ] OUT 2026-07-06T22:22:49.1002587Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:55.784713049Z [disk usage checker] Started [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:55.788001817Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.061259464Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.239660819Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.239693871Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.239693871Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.241943639Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.424054418Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.424084411Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.424084411Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.426149748Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.595802419Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.595834552Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.595834552Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.598023722Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.777735319Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.777769213Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.777769213Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.779872715Z Error getting logs from rhods-notebooks [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.960928064Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.960960495Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.960960495Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:56.963186503Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:57.134425708Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:57.134456752Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:57.134456752Z namespaces "istio-system" not found [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:57.136559357Z Error getting logs from istio-system [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:57.306951345Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:57.307053069Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:57.307053069Z namespaces "knative-serving" not found [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:57.309430649Z Error getting logs from knative-serving [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:57.482781130Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:57.482813460Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:57.482813460Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:57.484943077Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:57.791644610Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:58.098600211Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:58.242965951Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:58.329566484Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:58.331463663Z Error collecting info from [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:58.473385945Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:58.563641241Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:58.565774284Z Error collecting info from [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:58.713402377Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:58.800165369Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:58.801960176Z Error collecting info from [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:58.950618330Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:59.038668169Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:59.040675298Z Error collecting info from [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:59.187964369Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:59.273806140Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:59.275912222Z Error collecting info from [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:59.423254291Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:59.513457346Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:59.515737463Z Error collecting info from [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:59.660439075Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:59.745616609Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:59.747676223Z Error collecting info from [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:59.898444574Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:59.984359402Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:22:59.986529664Z Error collecting info from [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:00.129133168Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:00.219363584Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:00.221579312Z Error collecting info from [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:00.366792717Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:00.455629579Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:00.457855511Z Error collecting info from [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:00.598254073Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:00.686700170Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:00.688793953Z Error collecting info from [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:00.792483832Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:00.830125883Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:00.915744489Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:00.917830313Z Error collecting info from [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:01.058421062Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:01.146903777Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:01.148988020Z Error collecting info from [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:01.288154256Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:01.379028028Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:01.381299843Z Error collecting info from [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:01.530392940Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:01.619713775Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:01.621750351Z Error collecting info from [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:01.762817340Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:01.854144159Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:01.856421895Z Error collecting info from [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:01.999461948Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:02.086460741Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:02.088555608Z Error collecting info from [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:02.229335080Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:02.318108042Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:02.320174687Z Error collecting info from [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:03.076151152Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:03.213250573Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:03.354433337Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:03.488769408Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:03.627779776Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:04.249953649Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:04.386765992Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:04.528375084Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:04.681715823Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:04.818221343Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:05.392629264Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:05.719118119Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:05.799504510Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:06.429076488Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:06.803327597Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:06.975646824Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:07.266658370Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:07.988067512Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:08.345691095Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:08.517635667Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:08.804284919Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:09.506548861Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:09.852223377Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:10.023560140Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:10.340216299Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:10.804118937Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:11.092950929Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:11.444852618Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:11.624830832Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:11.928379751Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:12.694420982Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:13.081329685Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:13.263826369Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:13.578861744Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:14.322249606Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:14.669844059Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:14.845214207Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:15.187555158Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:15.808750118Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:15.922599175Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:16.274650263Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:16.450735324Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:16.783634092Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:17.504692651Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:17.847025267Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:18.017344301Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:18.317496790Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:19.013631334Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:19.356674289Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:19.528568809Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:19.821426005Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:20.515388293Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:20.813444740Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:20.904518697Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:21.076241259Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:21.371250512Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:22.070988436Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:22.415239257Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:22.594427480Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:22.887812166Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:23.583022882Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:23.927606717Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:24.100456922Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:24.405956769Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:25.103548657Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:25.464870893Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:25.639902048Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:25.820055317Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:25.939680272Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:26.643906838Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:27.022742038Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:27.197518978Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:27.520069545Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:28.276020124Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:28.627179696Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:28.799733128Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:29.121665410Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:29.844982664Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:30.186813267Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:30.358244490Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:30.825256691Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:31.690509303Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:31.865911398Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:32.243803619Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:32.415488637Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:33.795026727Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:33.973592843Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:34.359918920Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:34.534413986Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:35.832814152Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:35.899813105Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:36.057386801Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:36.448438955Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:36.619413520Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:38.000052507Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:38.160407910Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:38.542978553Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:38.715178752Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:39.269929243Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:39.464724114Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:39.859330225Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:40.027796449Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:40.581957572Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:40.778988929Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:40.837652369Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:41.174556407Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:41.345040299Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:41.937929887Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:42.123077007Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:42.526345168Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:42.702192246Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:43.251282194Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:43.439114507Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:43.844868029Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:44.015717280Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:44.611518715Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:44.810117614Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:45.209138792Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:45.376346192Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:45.845059390Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:45.923140479Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:46.112436645Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:46.507966827Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:46.677790274Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:47.307752503Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:47.503627876Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:47.889024473Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:48.061916080Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:48.605487467Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:48.789056278Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:49.181683529Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:49.354250418Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:49.905736167Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:50.105393555Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:50.502362445Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:50.677532403Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:50.850257109Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:51.232402164Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:51.424247805Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:51.821300117Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:51.991734492Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:52.542222622Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:52.733252698Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:53.133438035Z Wrote inspect data to must-gather. [must-gather] [must-gather-jb7l7] POD 2026-07-06T22:23:53.158679267Z Caches written to disk [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.166796328Z waiting for gather to complete [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.170615443Z downloading gather output [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.52506775Z receiving incremental file list [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.541504636Z ./ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.541586801Z aggregated-discovery-api.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.542070259Z aggregated-discovery-apis.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.543236448Z event-filter.html [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.545499961Z timestamp [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.545642219Z version [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.547726661Z cluster-scoped-resources/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.547763274Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.547798126Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.547838518Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.547974326Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.547995988Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.548029499Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.548154897Z namespaces/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.548185628Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.54821131Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.548336508Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.54837945Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.548441474Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.548479956Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.548575722Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.54889198Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.549137895Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.549183057Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.54922356Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.549276503Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.549321055Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.54939597Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.549451073Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.549492355Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.54957463Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.549622993Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.549665335Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.549820825Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.549923171Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.555236823Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.555283006Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.555652287Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.555792076Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.556076462Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.556161717Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.556196889Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.556296805Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.556344128Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.55637853Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.556404351Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.556486356Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.556528849Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.556566621Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.556639745Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.556704699Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.556757732Z namespaces/kserve-ci-e2e-test/pods/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.556766043Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-799f46c59b-pl8dd/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.556826236Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-799f46c59b-pl8dd/router-gateway-1-openshift-default-799f46c59b-pl8dd.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.556924032Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-799f46c59b-pl8dd/istio-proxy/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.556937673Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-799f46c59b-pl8dd/istio-proxy/istio-proxy/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.556946374Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-799f46c59b-pl8dd/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.556958604Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-799f46c59b-pl8dd/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.557193008Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-799f46c59b-pl8dd/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.557281943Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-54c789bdc6-22wcp/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.557327396Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-54c789bdc6-22wcp/router-gateway-2-openshift-default-54c789bdc6-22wcp.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.557419721Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-54c789bdc6-22wcp/istio-proxy/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.557430632Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-54c789bdc6-22wcp/istio-proxy/istio-proxy/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.557441172Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-54c789bdc6-22wcp/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.557468124Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-54c789bdc6-22wcp/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.557596062Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-54c789bdc6-22wcp/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.557662776Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-54c789bdc6-22wcp/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.557727919Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.557761981Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.557858607Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm/llm-d-routing-sidecar/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.557867357Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm/llm-d-routing-sidecar/llm-d-routing-sidecar/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.557872828Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.55791577Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/current.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.558175436Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/previous.insecure.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.55824582Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/previous.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.558282462Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm/main/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.558288912Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm/main/main/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.558294153Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm/main/main/logs/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.558318104Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm/main/main/logs/current.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.558436651Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm/main/main/logs/previous.insecure.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.558510575Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-wx8mm/main/main/logs/previous.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.558555228Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.55858582Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.558691656Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh/main/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.558705177Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh/main/main/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.558710677Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh/main/main/logs/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.558720248Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh/main/main/logs/current.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.558865446Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh/main/main/logs/previous.insecure.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.55893498Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-qbjsh/main/main/logs/previous.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.558971582Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.559011515Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.55910207Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb/main/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.559110621Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb/main/main/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.559115981Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb/main/main/logs/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.559149813Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb/main/main/logs/current.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.559790621Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb/main/main/logs/previous.insecure.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.559862005Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-599gxcb/main/main/logs/previous.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.559892456Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.559927619Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.559990342Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.560022474Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.560072867Z namespaces/kuadrant-system/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.560107229Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.560181113Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.560211565Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.560267319Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.560309981Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.560395436Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.560735856Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.561017533Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.561063065Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.561105488Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.56114082Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.561189763Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.561275678Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.56131829Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.561365423Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.561444048Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.561493201Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.561535023Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.561733485Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.561857722Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.562289197Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.562369872Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.562749744Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.562858781Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.563079064Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.563199491Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.563242284Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.56335506Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.563386712Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.563437175Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.563464546Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.563558462Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.563595564Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.563635116Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.56369875Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.563749653Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.563797866Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.563804977Z namespaces/kuadrant-system/pods/authorino-7c77957575-z4dhg/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.563843129Z namespaces/kuadrant-system/pods/authorino-7c77957575-z4dhg/authorino-7c77957575-z4dhg.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.563926354Z namespaces/kuadrant-system/pods/authorino-7c77957575-z4dhg/authorino/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.563936674Z namespaces/kuadrant-system/pods/authorino-7c77957575-z4dhg/authorino/authorino/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.563945755Z namespaces/kuadrant-system/pods/authorino-7c77957575-z4dhg/authorino/authorino/logs/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.563960506Z namespaces/kuadrant-system/pods/authorino-7c77957575-z4dhg/authorino/authorino/logs/current.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.567578158Z namespaces/kuadrant-system/pods/authorino-7c77957575-z4dhg/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.567650092Z namespaces/kuadrant-system/pods/authorino-7c77957575-z4dhg/authorino/authorino/logs/previous.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.567666724Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-gmblh/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.567715516Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-gmblh/authorino-operator-6d85f6564-gmblh.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.567838194Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-gmblh/manager/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.567853404Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-gmblh/manager/manager/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.567860725Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-gmblh/manager/manager/logs/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.567871936Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-gmblh/manager/manager/logs/current.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.568065677Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-gmblh/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.56812005Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-gmblh/manager/manager/logs/previous.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.568173063Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-wxznh/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.568186114Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-wxznh/dns-operator-controller-manager-65b49595d7-wxznh.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.568302101Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-wxznh/manager/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.568319182Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-wxznh/manager/manager/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.568330323Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-wxznh/manager/manager/logs/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.568346673Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-wxznh/manager/manager/logs/current.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.568445049Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-wxznh/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.568514633Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-wxznh/manager/manager/logs/previous.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.568540855Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7jl5x/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.568601779Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7jl5x/kuadrant-console-plugin-85fd4b8b9d-7jl5x.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.568701504Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7jl5x/kuadrant-console-plugin/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.568715135Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7jl5x/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.568723026Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7jl5x/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.568736316Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7jl5x/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.568826442Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7jl5x/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.568893556Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7jl5x/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.568935758Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.56897415Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.56913265Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw/manager/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.5691466Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw/manager/manager/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.569151231Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw/manager/manager/logs/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.569158491Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw/manager/manager/logs/current.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.587608435Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.587659658Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw/manager/manager/logs/previous.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.587747013Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-fsfzc/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.587768695Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-fsfzc/limitador-limitador-69574b596d-fsfzc.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.587904153Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-fsfzc/limitador/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.587913743Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-fsfzc/limitador/limitador/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.587919114Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-fsfzc/limitador/limitador/logs/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.587928324Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-fsfzc/limitador/limitador/logs/current.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.58803015Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-fsfzc/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.588114915Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-fsfzc/limitador/limitador/logs/previous.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.588127816Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-s2frj/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.588185639Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-s2frj/limitador-operator-controller-manager-6f9f468797-s2frj.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.588283345Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-s2frj/manager/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.588292125Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-s2frj/manager/manager/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.588297146Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-s2frj/manager/manager/logs/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.588327778Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-s2frj/manager/manager/logs/current.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.588487327Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-s2frj/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.588562711Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-s2frj/manager/manager/logs/previous.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.588598854Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.588645236Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.58870703Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.588743282Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.588811456Z namespaces/openshift-ingress/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.588856539Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.588925343Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.588959625Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.589014548Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.58905931Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.589148276Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.58939703Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.589626744Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.589665656Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.589722169Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.589803074Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.589842416Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.589928222Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.589967064Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.590015387Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.590096302Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.590145324Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.590191457Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.590635183Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.590759441Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.590979303Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.591057678Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.591330324Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.591457351Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.591696565Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.591796371Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.591839524Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.591923219Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.591960351Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.591992613Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.592053056Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.592132541Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.592180064Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.592212796Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.592275039Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.592308081Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.592363204Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.592370875Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.592399297Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/istiod-openshift-gateway-94bb8fbfd-zmx9k.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.592524804Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/discovery/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.592532934Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/discovery/discovery/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.592537505Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/discovery/discovery/logs/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.592574427Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/discovery/discovery/logs/current.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.634924825Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.63499644Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/discovery/discovery/logs/previous.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.635021791Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.635083405Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/openshift-ai-inference-openshift-default-9455cf8d5-vmnww.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.635222643Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/istio-proxy/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.635234364Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/istio-proxy/istio-proxy/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.635241104Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.635256335Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.635563643Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.635662709Z namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.635727472Z namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router-default-757d9b65f5-7xf7w.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.6358492Z namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.63585864Z namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router/router/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.635864051Z namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router/router/logs/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.635889222Z namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router/router/logs/current.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.63601924Z namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router/router/logs/previous.insecure.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.636093294Z namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router/router/logs/previous.log [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.636133116Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.636175079Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.636230552Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.636278655Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.643735533Z [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.643763205Z sent 3,366 bytes received 1,701,616 bytes 3,409,964.00 bytes/sec [must-gather] [must-gather-jb7l7] OUT 2026-07-06T22:23:54.643772945Z total size is 20,746,863 speedup is 12.17 [must-gather] [must-gather ] OUT 2026-07-06T22:23:54.895936082Z namespace/openshift-must-gather-b5t69 deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 45957e43-6e07-4af2-b514-0fb39f986eca [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.23" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-06T22:23:55.023296696Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba272a129f91bab6ed70a10b041d49836b63538fa2cbaa919e53a4a13fe3dcde [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 45957e43-6e07-4af2-b514-0fb39f986eca [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.23" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-06T22:23:55.044905946Z namespace/openshift-must-gather-8hm2m created [must-gather] [must-gather ] OUT 2026-07-06T22:23:55.055316047Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-bnsgb created [must-gather] [must-gather ] OUT 2026-07-06T22:23:55.077064185Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba272a129f91bab6ed70a10b041d49836b63538fa2cbaa919e53a4a13fe3dcde created [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:56.536488100Z [disk usage checker] Started [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:56.540087737Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:56.762039917Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:56.926174207Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:57.075420312Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:57.200362576Z Gathering data for ns/default... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:57.578997599Z Gathering data for ns/openshift... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:58.125902114Z Gathering data for ns/kube-system... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:58.937443898Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.321220302Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.321255063Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.321255063Z namespaces "assisted-installer" not found [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.645844782Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.653248672Z Executing Istio gather script [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.657080814Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.659123635Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.661600249Z INFO: Gathering HAProxy config files [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.666019969Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.683165052Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.689647986Z INFO: Collecting host service logs for crio [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.690011044Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.690323211Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.690616196Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.690951519Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.691300793Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.691633918Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.693016565Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.693354063Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.693688990Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.694033565Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:23:59.694433855Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:00.424360221Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:00.511030346Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:00.595942277Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:00.631188173Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:00.839550867Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:00.894100952Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-kjjpn [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:00.894152130Z insights-runtime-extractor-4ct7m [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:00.894171917Z insights-runtime-extractor-cmkcf [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:00.894189101Z insights-runtime-extractor-fl8wb [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:01.192103444Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:01.269328157Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:01.269458018Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:01.629714115Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:01.629714115Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:01.741092674Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:01.793165004Z No resources found [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:01.828803651Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:01.833815947Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:01.860349025Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:01.986229273Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:02.006681777Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:02.216313210Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:02.252341754Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:02.358743342Z No resources found [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:02.491724205Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:02.601326622Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:02.676575450Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:02.720120806Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:02.733898218Z INFO: 'previous-logs' folder not found on ip-10-0-131-45.ec2.internal, skipping... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:02.749129793Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:02.822557780Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:02.829279481Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:02.856410838Z tar: Removing leading `/' from member names [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:02.862195795Z INFO: HAProxy config collection complete. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:03.026672320Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:03.035446134Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:03.226357747Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:03.331249043Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:03.452440859Z INFO: 'previous-logs' folder not found on ip-10-0-131-57.ec2.internal, skipping... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:03.454687288Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:03.457964938Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:03.696038642Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:03.729168578Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:03.936424323Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:03.944450643Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:04.009986167Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:04.011830656Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:04.044645299Z INFO: 'previous-logs' folder not found on ip-10-0-142-77.ec2.internal, skipping... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:04.044690835Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:04.044703261Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:04.148306564Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:04.432370393Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:04.445562588Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:04.466251172Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:04.486084362Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:04.593811152Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:04.719556441Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:04.766978279Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:04.912863604Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:04.962911974Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:04.970383621Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.114170934Z INFO: INTERCONNECT MODE [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.114268553Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.146497649Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.161210092Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.301557516Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.305376764Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.350163726Z INFO: Gathering OVN_Northbound from ovnkube-node-82n9w... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.351617933Z INFO: Gathering OVN_Northbound from ovnkube-node-b2hcz... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.352539293Z INFO: Gathering OVN_Northbound from ovnkube-node-v7kmn... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.352899385Z INFO: Gathering OVN_Southbound from ovnkube-node-82n9w... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.353337817Z INFO: Gathering OVN_Southbound from ovnkube-node-b2hcz... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.353616255Z INFO: Gathering OVN_Southbound from ovnkube-node-v7kmn... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.469729144Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.589272672Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.700524107Z tar: Removing leading `/' from member names [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.779692197Z tar: Removing leading `/' from member names [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.800787725Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.805464102Z tar: Removing leading `/' from member names [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.811576873Z tar: Removing leading `/' from member names [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.813103664Z tar: Removing leading `/' from member names [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.836016610Z tar: Removing leading `/' from member names [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:05.889135090Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:06.006350800Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:06.199298226Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:06.384600320Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:06.444069394Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:06.444242869Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:06.444805882Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:06.446912207Z 7.7M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:06.449486797Z ovnk_database_store/ [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:06.449486797Z ovnk_database_store/ovnkube-node-82n9w_sbdb [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:06.496594715Z ovnk_database_store/ovnkube-node-b2hcz_sbdb [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:06.558980621Z ovnk_database_store/ovnkube-node-v7kmn_sbdb [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:06.605473261Z ovnk_database_store/ovnkube-node-82n9w_nbdb [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:06.619037861Z ovnk_database_store/ovnkube-node-v7kmn_nbdb [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:06.636296657Z ovnk_database_store/ovnkube-node-b2hcz_nbdb [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:06.664097501Z INFO: Network log collection complete. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:06.727332320Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:06.763309450Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:06.886067559Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:07.101892026Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:07.149446576Z W0706 22:24:07.149378 377 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:07.160191067Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:07.418222974Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:07.421430203Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:07.489412281Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:08.013918221Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:08.070053616Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:08.167922923Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:08.293146419Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:08.307314957Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:08.463012198Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:08.519049509Z Gathering data for ns/openshift-console... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:08.555874724Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:08.558530257Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:08.836333409Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:09.085480572Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:09.088190696Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:09.153638456Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:09.301538921Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:09.534227883Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:09.538043372Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:09.618443163Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:09.761209608Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:09.792305720Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:09.985003354Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:09.990536658Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:10.162150752Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:10.213971617Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:10.417648127Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:10.420654027Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:10.636705308Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:10.769957796Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:10.891812502Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:10.896448552Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:10.970904104Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:11.119657319Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:11.354549696Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:11.357715042Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:11.573069239Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:11.620392304Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:11.870493251Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:11.874096505Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:11.898627694Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:11.944591816Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:12.003914379Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:12.107554454Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:12.383305996Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:12.387626883Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:12.624828152Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:12.894512016Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:12.898314509Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:13.128425858Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:13.137749446Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:13.381679989Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:13.384415472Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:13.580443291Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:13.747940357Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:13.814278209Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:13.816807852Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:14.029724102Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:14.268530368Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:14.291221488Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:14.295134862Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:14.303723547Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:14.469301583Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:14.688721140Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:14.691042110Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:14.882915226Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:15.116383503Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:15.118856356Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:15.294396343Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:15.442948098Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:15.513492748Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:15.516515306Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:15.724413434Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:15.737340122Z Collecting performance related data for node ip-10-0-142-77.ec2.internal [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:15.745906043Z Collecting performance related data for node ip-10-0-131-45.ec2.internal [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:15.753581884Z Collecting performance related data for node ip-10-0-131-57.ec2.internal [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:15.979638553Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:15.983143465Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:16.192375591Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:16.475905038Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:16.479607510Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:16.731281100Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:16.904741948Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:17.035982028Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:17.039858981Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:17.293984129Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:17.561989781Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:17.565411693Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:17.700944711Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:17.828754175Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:18.070975600Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:18.073274990Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:18.285135531Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:18.369333864Z Gathering data for ns/kserve... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:18.526386207Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:18.530149053Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:18.741829453Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:18.985234533Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:18.987751275Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:19.226995091Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:19.480052371Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:19.630529697Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:19.863891893Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:20.910457385Z Gathering data for ns/openshift-config... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:21.324033199Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:21.327399773Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:21.515194161Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:21.642466915Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:21.790525782Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:21.915963555Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:21.949269501Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:22.060160263Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:22.350576855Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:22.778516370Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:23.332650709Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:23.464948837Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:23.783666201Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:23.847370878Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:23.966799815Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:24.218144823Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:24.500143142Z Done executing Istio gather script [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:24.637816291Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:24.807350872Z Collecting kubelet logs for node ip-10-0-142-77.ec2.internal [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:24.809198252Z Collecting kubelet logs for node ip-10-0-131-45.ec2.internal [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:24.811005288Z Collecting kubelet logs for node ip-10-0-131-57.ec2.internal [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:25.085832809Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:25.294004368Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-8hm2m namespace [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:25.299925712Z INFO: Node performance data collection complete. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:25.476485143Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:26.065891600Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:26.920967787Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:27.843250942Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:29.238291954Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:29.622642779Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:30.241813224Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:30.720796714Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:31.153233578Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:31.460935354Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:31.925589451Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:32.014205553Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:32.434439361Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:32.909832211Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:33.261224841Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:33.621132761Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:34.376913209Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:34.768310775Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:35.127268775Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:35.551244598Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:36.627990715Z Wrote inspect data to must-gather. [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:36.628043279Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:36.628043279Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:36.773221742Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:36.930530192Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 [must-gather] [must-gather-8t4g8] POD 2026-07-06T22:24:37.038465620Z Caches written to disk [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.050347116Z waiting for gather to complete [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.053458439Z downloading gather output [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.340020229Z receiving incremental file list [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.369087293Z ./ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.36919643Z aggregated-discovery-api.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.369600484Z aggregated-discovery-apis.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.370652756Z event-filter.html [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.373138232Z timestamp [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.373216727Z version [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.380596272Z cluster-scoped-resources/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.380618744Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.380633474Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.380771033Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.380943443Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.381125583Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.381210778Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.381308924Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.381445382Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.381535238Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.381667395Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.381726569Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.381851246Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.38192537Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.382413659Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.382444421Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.382653723Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.382923759Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.383137712Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.383285241Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.38666951Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.388349119Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.388481207Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.388598774Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.388735742Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.38886871Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.38937166Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.389510968Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.389674078Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.389818956Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.390583881Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.390842166Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.390964584Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.391129983Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.392280021Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.392499404Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.392689536Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.392859075Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.393818282Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.393980061Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.394216565Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.394581287Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.395828641Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.395971979Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.396108037Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.396244345Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.396354081Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.396466078Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.396643578Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.396853841Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.397616546Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.397766565Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.397897932Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.399778314Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.399945673Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.400096372Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.400329176Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.400622433Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.401055079Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.401168816Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.401299353Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.40140868Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.401518736Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.401644304Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.401770101Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.401900349Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.402039347Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.402148453Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.402405489Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.402551947Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.402662184Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.403224037Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.403341533Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.40346034Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.403732827Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.404482451Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.404604978Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.40481458Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.404925027Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.405088437Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.405208504Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.405360543Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.40548796Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.405613108Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.405750016Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.405872933Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.405981929Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.406124818Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.406265876Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.406406744Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.406548683Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.406700282Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.407070973Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.407239563Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.407613276Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.407759124Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.40853259Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.408676218Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.408796005Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.408911152Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.409031329Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.40922396Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.409468695Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.409599093Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.409731241Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.40989321Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.410017567Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.410132114Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepoolimports.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.410269682Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.41041127Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.410554569Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.412835413Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.413258969Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.413860174Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.414032204Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.414194224Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.414332372Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.41446626Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.414574846Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.414706424Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.41548308Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.415819009Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.418783594Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.418890801Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.419062731Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.423063457Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.423218466Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.423334773Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.423465971Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.42633495Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.426598085Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.433544115Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.437970506Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.438126435Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.438420893Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.438590152Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.438771083Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.43889064Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.439046639Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.439345847Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.439424602Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.439533828Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.439750861Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.439882389Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.439998275Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.440110892Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.440222409Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.440364417Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.440505965Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.440669575Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.440809763Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.440955192Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.441107611Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.441228888Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.441393558Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.441524876Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.441675274Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.442003834Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.442160293Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.442425569Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.442539176Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.442656472Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.44278305Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.444898444Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.445017591Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.445132988Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.445244645Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.445344201Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.44550001Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.445620987Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.445798218Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.445924245Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.447662027Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.447849249Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.447981086Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.448118785Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.448239562Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.448434283Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.448704819Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.449265902Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.449555699Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.449674856Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.449816765Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.449926431Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.4505924Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.450804263Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.450935601Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.45245238Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.452612339Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.452802801Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.452911897Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.453073617Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.453352793Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.453479741Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.4536486Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.454083696Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.454189003Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.454326751Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.454496931Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.45465207Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.454810699Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.454954568Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.455213093Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.455524001Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.455571164Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.455662499Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.455797837Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.455881762Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.455966517Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.456122506Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.456226483Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.456312568Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.456403643Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.456490558Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.456595925Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.456707041Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.456798906Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.456890292Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.456978147Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.457068582Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.457154967Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.457244143Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.457335568Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.457419803Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.457506158Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.457596953Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.45770082Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.457809546Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.457893851Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.457980716Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.458060651Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.458157957Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.458243342Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.458329587Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.458427142Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.458513487Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.458599003Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.458705049Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.458788934Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.458881689Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.458964784Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.45905909Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.459144695Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.45923119Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.459322545Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.459420041Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.459495656Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.459585931Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.459676116Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.459786273Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.459872348Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.459985704Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.46007888Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.460173625Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.460260731Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.460353666Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.460442711Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.460536027Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.460638123Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.46075189Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.460838615Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.460941271Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.461031936Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.461126892Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.461225218Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.461314823Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.461405788Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.461504754Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.46161171Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.461724717Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.461826493Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.461915068Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.462008884Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.462097049Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.462191715Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.46227966Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.462375025Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.46246372Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.462559326Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.462660862Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.462772949Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.462866964Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.46296846Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.463064106Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.463145271Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.463239316Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.463327231Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.463422747Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.463510522Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.463597827Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.463707164Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.463831921Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.463919386Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.464011342Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.464107717Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.464213514Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.464297589Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.464391804Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.464474419Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.464576385Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.464698922Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.464795078Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.464875173Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.464965548Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.465056004Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.465152159Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.465241164Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.46533345Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.465420245Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.46551065Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.465566834Z cluster-scoped-resources/certificates.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.465577604Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.465607836Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-6mzhr.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.465761635Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-bwbgq.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.465860851Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-vkv4v.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.466047722Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.466090085Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.466260375Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.466407413Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.466496018Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.466592294Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.466942465Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.467052791Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.467138116Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.467240772Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.467344348Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.467427153Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.467520249Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.467611464Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.467723911Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.467821307Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.467926473Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.468018628Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.468124194Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.468204609Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.468299325Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.46839578Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.468496576Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.468582641Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.468752611Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.468814645Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.468931842Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.469033008Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.469139134Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.469223339Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.469327255Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.46941443Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.46957978Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.469675396Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.469788623Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.469878378Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.469978094Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.470112042Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.470209317Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.470303633Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.470400829Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.470502475Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.47059884Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.470706027Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.470819673Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.470914649Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.470988133Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.471039466Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.471118651Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.471158683Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.471225967Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.471257869Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.471333594Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.471374076Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.471415948Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.471500904Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.471594849Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.471671714Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.471848104Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.471868955Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.47194955Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.472044105Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.472152622Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.472206325Z cluster-scoped-resources/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.472219576Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.472269879Z cluster-scoped-resources/core/nodes/ip-10-0-131-45.ec2.internal.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.472473431Z cluster-scoped-resources/core/nodes/ip-10-0-131-57.ec2.internal.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.472640351Z cluster-scoped-resources/core/nodes/ip-10-0-142-77.ec2.internal.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.47279032Z cluster-scoped-resources/core/persistentvolumes/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.472838492Z cluster-scoped-resources/core/persistentvolumes/pvc-cff970c0-1328-4734-8bb0-45cf29b3a727.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.472908027Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.472985131Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.473032234Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.47312927Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.473223285Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.47330494Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.473394165Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.47347721Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.473632449Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.473762957Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.473873813Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.473971759Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.474067425Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.474169191Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.474250836Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.474341801Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.474435467Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.474529862Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.474625558Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.474749195Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.47483157Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.474931506Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.475015381Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.475096116Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.475137648Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.475231754Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.475314969Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.475408224Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.475492439Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.475588425Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.475765795Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.475917964Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.476007399Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.476074023Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.476082904Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.476127177Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.476203291Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.476212262Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.476243843Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.476324748Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.476367671Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.476429874Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.476440575Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.476491168Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.476618335Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.476844409Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.476906912Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.476915773Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.476962386Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.47703377Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.47704147Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.477082693Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.477156157Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.4772109Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.477314567Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.477358019Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.477438174Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.477480286Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.47754894Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.477590303Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.477704049Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.477762733Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.477826917Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.477863379Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.477915212Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.477955324Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.478017288Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.47805743Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.478135025Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.478175787Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.478255562Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.478296014Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.478335767Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.478376229Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.478417712Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.478457644Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.478515417Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.47855657Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.478651475Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.478659036Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.478703628Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.478808195Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.478855777Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.478977165Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.479282763Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.479446532Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.479555749Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.479673796Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.47991836Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.480026276Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.48008313Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.480135013Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.480221658Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.480307823Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.480469513Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.480616761Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.480747849Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.480841975Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.480926749Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.481022315Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.48110864Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.481200456Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.481294961Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.481396937Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.481469081Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.481559027Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.481650502Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.481776619Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.481863545Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.48196255Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.482049496Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.482152762Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.482251078Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.48228803Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.482397456Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.482478491Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.482569576Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.482742817Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.482844743Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.482932488Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.483026313Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.483118569Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.483202084Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.483291369Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.483385974Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.48347361Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.483563725Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.483659781Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.483784048Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.483880864Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.483966719Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.484062504Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.4841556Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.484245825Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.484355122Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.484441117Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.484535672Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.484650119Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.484765476Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.484867942Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.484956807Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.485046323Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.485144648Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.485197251Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.485208092Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.485243454Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.485303258Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.485311858Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.48534077Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.485416814Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.485424645Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.485478558Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.485538561Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.485578024Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-131-45.ec2.internal.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.485698461Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-131-57.ec2.internal.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.485792256Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-142-77.ec2.internal.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.48585513Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.485898453Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.485985738Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.486049512Z host_service_logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.486061872Z host_service_logs/masters/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.486112555Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.48619296Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.486263254Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.486331558Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.486481067Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.486599494Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.486672128Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.486769594Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.486851599Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.486936714Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.486993527Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.48703684Z ingress_controllers/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.487047121Z ingress_controllers/default/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.487051961Z ingress_controllers/default/router-default-757d9b65f5-7xf7w/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.487112744Z ingress_controllers/default/router-default-757d9b65f5-7xf7w/haproxy.config [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.487227111Z insights-data/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.487238652Z istio/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.487295035Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.487419792Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.487953364Z istio/event-filter.html [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.488103993Z istio/timestamp [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.488239881Z istio/cluster-scoped-resources/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.488247711Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.488254172Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.488301444Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.488384509Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.488458414Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.488520627Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.488615583Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.488667526Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.488959193Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.489842495Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.490058048Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.490220828Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.490616481Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.490835644Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.491305041Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.492108909Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.492245287Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.492377455Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepoolimports.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.492516113Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.492668002Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.49280795Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.492934198Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.493036284Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.49315315Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.493304119Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.49347804Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.493770337Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.493945057Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.49433168Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.494461728Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.494602586Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.494758295Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.494772236Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.494822269Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.494895853Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.494903154Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.494939686Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.495034852Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.495193931Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.495261725Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.495305037Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.495402113Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.495575224Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.495635777Z istio/namespaces/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.495658118Z istio/namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.49569117Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.495711321Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.495763344Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.49585985Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-2.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.495948326Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.495996178Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.496091934Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-2.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.496175099Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-3.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.496265904Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-4.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.49636566Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/scheduler-v06-pd-migration-test-kserve-route.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.496498878Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.496509948Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.49654301Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/scheduler-v06-pd-migration-test-inference-pool.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.496621065Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.496632246Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.496674618Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/scheduler-v06-pd-migration-test-inference-pool.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.496827627Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.496848519Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.496898282Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/scheduler-v06-pd-migration-test-kserve-scheduler.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.497000728Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/scheduler-v06-pd-migration-test-kserve-shadow-svc.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.497158857Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/scheduler-v06-pd-migration-test-kserve-workload-svc.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.497244602Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.497314086Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.497424062Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.497478286Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.49755644Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.497590902Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.497662576Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.497935223Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.498133524Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.498184417Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.498215979Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.498294794Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.498328926Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.498423301Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.498478685Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.498535408Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.498616243Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.498702748Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.49873992Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.499326735Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.49942423Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.499651844Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.49975797Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.500004025Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.500127422Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.500350835Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.50043629Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.500484903Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.500560898Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.500578728Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.500610001Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.500700636Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.50076319Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.500834864Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.500872076Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.500964711Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.501046216Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.501092349Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.501144442Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.501198855Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.501216386Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.501290961Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.501403717Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.501505173Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.50162903Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.501672073Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.501771009Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.50178355Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.501825062Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/istiod-openshift-gateway-94bb8fbfd-zmx9k.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.501930828Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/discovery/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.501940879Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/discovery/discovery/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.501953009Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/discovery/discovery/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.501989742Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/discovery/discovery/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.524421775Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.52450542Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/discovery/discovery/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.524582274Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.524637467Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/openshift-ai-inference-openshift-default-9455cf8d5-vmnww.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.524771885Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/istio-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.524794807Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/istio-proxy/istio-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.524805067Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.524862891Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.525154008Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.525252024Z istio/namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.525292506Z istio/namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router-default-757d9b65f5-7xf7w.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.525424184Z istio/namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.525437665Z istio/namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router/router/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.525450115Z istio/namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router/router/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.525483868Z istio/namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router/router/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.525596514Z istio/namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router/router/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.525667218Z istio/namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router/router/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.525724492Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.525766414Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.525809417Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.525842609Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.525884761Z machine_config_ondisk/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.525892272Z machine_config_termination_logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.525899642Z monitoring/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.525906702Z monitoring/alertmanager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.525963916Z monitoring/alertmanager/status.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.526062802Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.526127855Z monitoring/prometheus/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.526167858Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.526266183Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.526331518Z monitoring/prometheus/rules.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.527015848Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.52705505Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.527107703Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.528043588Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.528108272Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.528155095Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.528251791Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.528319935Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.52840996Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.528422661Z monitoring/prometheus/status/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.528479304Z monitoring/prometheus/status/config.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.528975343Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.529044918Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.529140923Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.529365506Z namespaces/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.529375857Z namespaces/cert-manager-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.529381357Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.529386307Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.529411829Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.529483273Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.529496774Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.529544007Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.52975934Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.530008494Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.530166504Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.530387326Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.53061083Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.530649282Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-fdrh7.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.530826632Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.530863635Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.530939769Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.530963171Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.531041135Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.531073597Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.531157892Z namespaces/cert-manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.531171003Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.531181184Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.531216026Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.531376575Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.531573397Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.531722355Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.531942998Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.532164311Z namespaces/default/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.532197113Z namespaces/default/default.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.532278428Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.53231083Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.532391305Z namespaces/default/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.532420826Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.532508292Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.532589796Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.532671941Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.532742696Z namespaces/default/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.532782448Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.53282725Z namespaces/default/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.532874113Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.532953778Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.533014972Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.533055054Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.533146119Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.533233334Z namespaces/default/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.533269407Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.533384303Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.533467208Z namespaces/default/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.533698992Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.533786537Z namespaces/default/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.533891893Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.534057223Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.534253105Z namespaces/default/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.534326359Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.534363781Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.534447306Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.534487708Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.534569603Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.534614316Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.534708661Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.534794967Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.534837349Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.534881602Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.534910953Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.534973027Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.535012819Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.53502422Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.535056742Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.535312957Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.535491588Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.535636176Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.535854329Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.53603929Z namespaces/default/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.536071322Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.536124645Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.536166438Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.53621205Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.536219401Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.536226721Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.536263853Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/epp-kserve-ci-e2e-test-scheduler-ha-replicas-test-inference-pool.llm-d.ai.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.536340438Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.536349208Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.536392141Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.536487276Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-2.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.536564551Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.536583232Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.536639785Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.536756942Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.536816506Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.536857768Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.536957794Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.537005757Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.537017108Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.537069281Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.53721914Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.53740288Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.537534878Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.537774412Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.538032737Z namespaces/kserve/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.538091661Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.538206648Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.538269791Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.538355937Z namespaces/kserve/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.538393169Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.538489564Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.538701787Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.53891706Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.538940271Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.539002545Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.539037577Z namespaces/kserve/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.53908313Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.539163244Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.539248229Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.53926712Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.539363786Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.539410099Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.539418629Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.53942924Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.539519375Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.539609961Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.539744958Z namespaces/kserve/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.539786861Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.540028255Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.540140802Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.54045142Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.540528885Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.540851854Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.540985952Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.541243287Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.541331242Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.541368614Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.541496902Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.541517183Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.541603438Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.54164115Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.541776748Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.541860563Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.541922037Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.54196795Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.542025183Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.542082106Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.542166491Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.542201143Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.542245416Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.54231007Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.54231995Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.542371904Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.542564775Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.542786018Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.542912275Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.543111377Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.54333407Z namespaces/kserve/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.543342551Z namespaces/kserve/pods/kserve-controller-manager-68bc99fcb8-7bghc/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.543397314Z namespaces/kserve/pods/kserve-controller-manager-68bc99fcb8-7bghc/kserve-controller-manager-68bc99fcb8-7bghc.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.54350143Z namespaces/kserve/pods/kserve-controller-manager-68bc99fcb8-7bghc/manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.543513681Z namespaces/kserve/pods/kserve-controller-manager-68bc99fcb8-7bghc/manager/manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.543531872Z namespaces/kserve/pods/kserve-controller-manager-68bc99fcb8-7bghc/manager/manager/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.543594216Z namespaces/kserve/pods/kserve-controller-manager-68bc99fcb8-7bghc/manager/manager/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.543737634Z namespaces/kserve/pods/kserve-controller-manager-68bc99fcb8-7bghc/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.543814298Z namespaces/kserve/pods/kserve-controller-manager-68bc99fcb8-7bghc/manager/manager/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.543853961Z namespaces/kserve/pods/llmisvc-controller-manager-555fd59bd5-82brg/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.543921755Z namespaces/kserve/pods/llmisvc-controller-manager-555fd59bd5-82brg/llmisvc-controller-manager-555fd59bd5-82brg.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.54401388Z namespaces/kserve/pods/llmisvc-controller-manager-555fd59bd5-82brg/manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.544030141Z namespaces/kserve/pods/llmisvc-controller-manager-555fd59bd5-82brg/manager/manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.544044842Z namespaces/kserve/pods/llmisvc-controller-manager-555fd59bd5-82brg/manager/manager/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.544083174Z namespaces/kserve/pods/llmisvc-controller-manager-555fd59bd5-82brg/manager/manager/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.547205029Z namespaces/kserve/pods/llmisvc-controller-manager-555fd59bd5-82brg/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.547294324Z namespaces/kserve/pods/llmisvc-controller-manager-555fd59bd5-82brg/manager/manager/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.547357038Z namespaces/kserve/pods/model-serving-api-fd65d7d6-s8blt/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.547416511Z namespaces/kserve/pods/model-serving-api-fd65d7d6-s8blt/model-serving-api-fd65d7d6-s8blt.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.547524547Z namespaces/kserve/pods/model-serving-api-fd65d7d6-s8blt/server/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.547534188Z namespaces/kserve/pods/model-serving-api-fd65d7d6-s8blt/server/server/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.547544118Z namespaces/kserve/pods/model-serving-api-fd65d7d6-s8blt/server/server/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.547613983Z namespaces/kserve/pods/model-serving-api-fd65d7d6-s8blt/server/server/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.54774831Z namespaces/kserve/pods/model-serving-api-fd65d7d6-s8blt/server/server/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.547826095Z namespaces/kserve/pods/model-serving-api-fd65d7d6-s8blt/server/server/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.547875398Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-zlrl9/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.547919571Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-zlrl9/odh-model-controller-84b54b7f97-zlrl9.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.548009976Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-zlrl9/manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.548021827Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-zlrl9/manager/manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.548027697Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-zlrl9/manager/manager/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.548099001Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-zlrl9/manager/manager/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.576035479Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-zlrl9/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.576106313Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-zlrl9/manager/manager/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.576127094Z namespaces/kserve/pods/s3-init-vmx7n/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.576202989Z namespaces/kserve/pods/s3-init-vmx7n/s3-init-vmx7n.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.576291434Z namespaces/kserve/pods/s3-init-vmx7n/download-hf-model/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.576309235Z namespaces/kserve/pods/s3-init-vmx7n/download-hf-model/download-hf-model/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.576316035Z namespaces/kserve/pods/s3-init-vmx7n/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.576343827Z namespaces/kserve/pods/s3-init-vmx7n/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.576432282Z namespaces/kserve/pods/s3-init-vmx7n/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.576515767Z namespaces/kserve/pods/s3-init-vmx7n/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.576529318Z namespaces/kserve/pods/s3-init-vmx7n/s3-init/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.576535168Z namespaces/kserve/pods/s3-init-vmx7n/s3-init/s3-init/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.576580281Z namespaces/kserve/pods/s3-init-vmx7n/s3-init/s3-init/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.576636434Z namespaces/kserve/pods/s3-init-vmx7n/s3-init/s3-init/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.576996995Z namespaces/kserve/pods/s3-init-vmx7n/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.57706601Z namespaces/kserve/pods/s3-init-vmx7n/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.577086091Z namespaces/kserve/pods/seaweedfs-64568bcd49-28mhv/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.577160705Z namespaces/kserve/pods/seaweedfs-64568bcd49-28mhv/seaweedfs-64568bcd49-28mhv.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.577278232Z namespaces/kserve/pods/seaweedfs-64568bcd49-28mhv/seaweedfs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.577289113Z namespaces/kserve/pods/seaweedfs-64568bcd49-28mhv/seaweedfs/seaweedfs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.577298563Z namespaces/kserve/pods/seaweedfs-64568bcd49-28mhv/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.577385858Z namespaces/kserve/pods/seaweedfs-64568bcd49-28mhv/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.577624753Z namespaces/kserve/pods/seaweedfs-64568bcd49-28mhv/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.577699497Z namespaces/kserve/pods/seaweedfs-64568bcd49-28mhv/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.577759761Z namespaces/kserve/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.577798583Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.577869227Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.577907879Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.578040657Z namespaces/kuadrant-system/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.57809193Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.578172715Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.578204327Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.57826169Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.578300372Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.578391428Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.578746099Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.579031286Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.579073358Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.579132212Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.579201436Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.579241868Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.579329963Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.579367505Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.579418018Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.579496983Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.579531015Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.579538356Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.579587768Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.579674084Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.57978525Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.579865535Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.579899117Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.57995409Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.580110829Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.580230196Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.580764968Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.580841412Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.581201744Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.581365713Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.581624339Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.581774257Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.58182505Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.581946187Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.58199123Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.582043033Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.582090866Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.582172531Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.582252285Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.582304689Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.582344911Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.582397814Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.582440607Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.582509381Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.582519191Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.582571804Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.582798678Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.583031191Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.58318151Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.583397193Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.583616646Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.583648648Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-9cq54.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.583987118Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.584035861Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.584130496Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.584221452Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.584298726Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.58436022Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.584408383Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.584472647Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.584518989Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.584633116Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.584815897Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.584922563Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.585009998Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.585020889Z namespaces/kuadrant-system/pods/authorino-7c77957575-z4dhg/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.585056031Z namespaces/kuadrant-system/pods/authorino-7c77957575-z4dhg/authorino-7c77957575-z4dhg.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.585134226Z namespaces/kuadrant-system/pods/authorino-7c77957575-z4dhg/authorino/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.585143966Z namespaces/kuadrant-system/pods/authorino-7c77957575-z4dhg/authorino/authorino/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.585153267Z namespaces/kuadrant-system/pods/authorino-7c77957575-z4dhg/authorino/authorino/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.585199749Z namespaces/kuadrant-system/pods/authorino-7c77957575-z4dhg/authorino/authorino/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.588914209Z namespaces/kuadrant-system/pods/authorino-7c77957575-z4dhg/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.588968962Z namespaces/kuadrant-system/pods/authorino-7c77957575-z4dhg/authorino/authorino/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.589038356Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-gmblh/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.589074368Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-gmblh/authorino-operator-6d85f6564-gmblh.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.589177764Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-gmblh/manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.589190165Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-gmblh/manager/manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.589199225Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-gmblh/manager/manager/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.589233197Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-gmblh/manager/manager/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.589410388Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-gmblh/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.589477882Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-gmblh/manager/manager/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.589535495Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-wxznh/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.589566627Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-wxznh/dns-operator-controller-manager-65b49595d7-wxznh.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.589658722Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-wxznh/manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.589667053Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-wxznh/manager/manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.589700785Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-wxznh/manager/manager/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.589755248Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-wxznh/manager/manager/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.589861214Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-wxznh/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.589935429Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-wxznh/manager/manager/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.58995199Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7jl5x/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.590006493Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7jl5x/kuadrant-console-plugin-85fd4b8b9d-7jl5x.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.590087228Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7jl5x/kuadrant-console-plugin/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.590095878Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7jl5x/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.590100808Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7jl5x/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.590135281Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7jl5x/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.590212085Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7jl5x/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.590279449Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7jl5x/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.59029661Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.590355574Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.59046081Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw/manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.5904713Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw/manager/manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.590475701Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw/manager/manager/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.590499202Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw/manager/manager/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.602493409Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.602544712Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-lg8kw/manager/manager/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.602598516Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-fsfzc/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.602641448Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-fsfzc/limitador-limitador-69574b596d-fsfzc.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.602785817Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-fsfzc/limitador/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.602799197Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-fsfzc/limitador/limitador/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.602804378Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-fsfzc/limitador/limitador/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.602831229Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-fsfzc/limitador/limitador/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.602931015Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-fsfzc/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.60302265Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-fsfzc/limitador/limitador/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.603060543Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-s2frj/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.603121116Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-s2frj/limitador-operator-controller-manager-6f9f468797-s2frj.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.603215232Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-s2frj/manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.603225193Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-s2frj/manager/manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.603230693Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-s2frj/manager/manager/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.603258535Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-s2frj/manager/manager/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.603414504Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-s2frj/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.603476087Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-s2frj/manager/manager/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.603505159Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.603552732Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.603604695Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.603632566Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.60369458Z namespaces/kube-node-lease/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.603709061Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.603733092Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.603806427Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-131-45.ec2.internal.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.603894492Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-131-57.ec2.internal.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.603981937Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-142-77.ec2.internal.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.60402919Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.604047651Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.604106285Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.60454188Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.604789055Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.604935983Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.605251162Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.605467145Z namespaces/kube-public/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.605477795Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.605487626Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.605525888Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.605706109Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.60589062Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.606055789Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.606265422Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.606524767Z namespaces/kube-system/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.606561109Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.606629643Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.606671666Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.606772102Z namespaces/kube-system/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.606793873Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.606933371Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.607017716Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.607099861Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.607135873Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.607181866Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.607216998Z namespaces/kube-system/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.60725032Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.607344876Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.607388578Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.60742663Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.607502425Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.607547807Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.607562968Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.607608031Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-t3dhh6xwfqkbmnkfk6btrshvxm.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.607708587Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.607812253Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.607891698Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.608037736Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.608116211Z namespaces/kube-system/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.608150163Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.608308162Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.608388017Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.608674014Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.608792651Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.609072337Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.609206635Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.609573347Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.609642441Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.609696904Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.609809151Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.609839983Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.609891686Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.609922997Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.610012263Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.610086107Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.61013151Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.610169082Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.610217955Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.610256567Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.6102978Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.610328701Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.610364254Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.610520853Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.610727095Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.610866693Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.611071445Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.611279388Z namespaces/kube-system/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.611288778Z namespaces/kube-system/pods/global-pull-secret-syncer-4r7cf/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.611345352Z namespaces/kube-system/pods/global-pull-secret-syncer-4r7cf/global-pull-secret-syncer-4r7cf.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.611429726Z namespaces/kube-system/pods/global-pull-secret-syncer-4r7cf/global-pull-secret-syncer/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.611444417Z namespaces/kube-system/pods/global-pull-secret-syncer-4r7cf/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.611450008Z namespaces/kube-system/pods/global-pull-secret-syncer-4r7cf/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.611460268Z namespaces/kube-system/pods/global-pull-secret-syncer-4r7cf/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.611856071Z namespaces/kube-system/pods/global-pull-secret-syncer-4r7cf/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.611923575Z namespaces/kube-system/pods/global-pull-secret-syncer-4r7cf/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.611953147Z namespaces/kube-system/pods/global-pull-secret-syncer-dnd82/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.611991719Z namespaces/kube-system/pods/global-pull-secret-syncer-dnd82/global-pull-secret-syncer-dnd82.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.612102806Z namespaces/kube-system/pods/global-pull-secret-syncer-dnd82/global-pull-secret-syncer/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.612115927Z namespaces/kube-system/pods/global-pull-secret-syncer-dnd82/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.612121457Z namespaces/kube-system/pods/global-pull-secret-syncer-dnd82/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.612139368Z namespaces/kube-system/pods/global-pull-secret-syncer-dnd82/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.6125027Z namespaces/kube-system/pods/global-pull-secret-syncer-dnd82/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.612573994Z namespaces/kube-system/pods/global-pull-secret-syncer-dnd82/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.612617667Z namespaces/kube-system/pods/global-pull-secret-syncer-w86tz/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.612655619Z namespaces/kube-system/pods/global-pull-secret-syncer-w86tz/global-pull-secret-syncer-w86tz.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.612748644Z namespaces/kube-system/pods/global-pull-secret-syncer-w86tz/global-pull-secret-syncer/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.612769745Z namespaces/kube-system/pods/global-pull-secret-syncer-w86tz/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.612774676Z namespaces/kube-system/pods/global-pull-secret-syncer-w86tz/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.612822899Z namespaces/kube-system/pods/global-pull-secret-syncer-w86tz/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.613220742Z namespaces/kube-system/pods/global-pull-secret-syncer-w86tz/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.613281575Z namespaces/kube-system/pods/global-pull-secret-syncer-w86tz/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.613323388Z namespaces/kube-system/pods/konnectivity-agent-5hpfr/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.613370871Z namespaces/kube-system/pods/konnectivity-agent-5hpfr/konnectivity-agent-5hpfr.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.613469717Z namespaces/kube-system/pods/konnectivity-agent-5hpfr/konnectivity-agent/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.613481857Z namespaces/kube-system/pods/konnectivity-agent-5hpfr/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.613493338Z namespaces/kube-system/pods/konnectivity-agent-5hpfr/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.61352525Z namespaces/kube-system/pods/konnectivity-agent-5hpfr/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.614516788Z namespaces/kube-system/pods/konnectivity-agent-5hpfr/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.614593263Z namespaces/kube-system/pods/konnectivity-agent-5hpfr/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.614648006Z namespaces/kube-system/pods/konnectivity-agent-n6892/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.614700119Z namespaces/kube-system/pods/konnectivity-agent-n6892/konnectivity-agent-n6892.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.614783084Z namespaces/kube-system/pods/konnectivity-agent-n6892/konnectivity-agent/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.614790925Z namespaces/kube-system/pods/konnectivity-agent-n6892/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.614795445Z namespaces/kube-system/pods/konnectivity-agent-n6892/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.614834567Z namespaces/kube-system/pods/konnectivity-agent-n6892/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.615943953Z namespaces/kube-system/pods/konnectivity-agent-n6892/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.616014867Z namespaces/kube-system/pods/konnectivity-agent-n6892/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.616044049Z namespaces/kube-system/pods/konnectivity-agent-zbdwk/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.616091791Z namespaces/kube-system/pods/konnectivity-agent-zbdwk/konnectivity-agent-zbdwk.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.616168986Z namespaces/kube-system/pods/konnectivity-agent-zbdwk/konnectivity-agent/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.616176986Z namespaces/kube-system/pods/konnectivity-agent-zbdwk/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.616183287Z namespaces/kube-system/pods/konnectivity-agent-zbdwk/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.61624561Z namespaces/kube-system/pods/konnectivity-agent-zbdwk/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.617234129Z namespaces/kube-system/pods/konnectivity-agent-zbdwk/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.617304363Z namespaces/kube-system/pods/konnectivity-agent-zbdwk/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.617349046Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.617402949Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.617487163Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal/haproxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.617497074Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.617503544Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.617546237Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.619993442Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.620050785Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-45.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.620112098Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.62013403Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.620230025Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal/haproxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.620239786Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.620244956Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.620270888Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.620595247Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.62065609Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.620666811Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.620801119Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.620863863Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal/haproxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.620869443Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.620873333Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.620936377Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.621232375Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.621312389Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.62133435Z namespaces/kube-system/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.621381973Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.621435307Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.621443397Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.621505501Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.621617967Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.621672721Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.621806228Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.62183197Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.621884463Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.621927616Z namespaces/open-cluster-management-708a374e-9898-4f03-b4f0-15036756e/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.621940076Z namespaces/open-cluster-management-708a374e-9898-4f03-b4f0-15036756e/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.621952687Z namespaces/open-cluster-management-708a374e-9898-4f03-b4f0-15036756e/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.62200806Z namespaces/open-cluster-management-708a374e-9898-4f03-b4f0-15036756e/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.622153519Z namespaces/open-cluster-management-708a374e-9898-4f03-b4f0-15036756e/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.622351981Z namespaces/open-cluster-management-708a374e-9898-4f03-b4f0-15036756e/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.622490509Z namespaces/open-cluster-management-708a374e-9898-4f03-b4f0-15036756e/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.622709722Z namespaces/open-cluster-management-708a374e-9898-4f03-b4f0-15036756e/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.622939665Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.622950926Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.622956306Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.622970037Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.623064672Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.623116066Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.623150848Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.62319017Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.623410503Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.623602804Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.623887131Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.624129205Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.624357839Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.624418763Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.624491937Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.62454886Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.624615084Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.624642196Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.624770343Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.624852298Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.624934863Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.624978125Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.625006047Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.625064221Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.625114573Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.625190108Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.625233131Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.625271683Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.625357948Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.625413591Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.625469454Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.63218451Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.632288536Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.632389842Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.632474958Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.632559263Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.632709911Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.632910833Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.633056332Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.633104855Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.633165838Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.633208821Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.633265164Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.633302116Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.633398752Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.633472877Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.6335325Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.633567912Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.633640686Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.63369621Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.633784365Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.633814057Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.633885451Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.633895262Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.633924253Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.634091613Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.634280794Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.634421232Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.634728371Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.634939503Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.634970065Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.635023378Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.63506301Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.635159276Z namespaces/openshift-apiserver/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.635207629Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.635275323Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.635315935Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.635363828Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.6354061Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.635488385Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.6355663Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.635645954Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.635700838Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.635762821Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.635831856Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.635878328Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.635962633Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.636039228Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.63608424Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.636169496Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.63624967Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.636301983Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.63641142Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.636491204Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.6365806Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.636663255Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.636764761Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.636910149Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.63708715Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.637158964Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.637208167Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.63726593Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.637303643Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.637365466Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.637413509Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.637540286Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.637640632Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.637696626Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.637789961Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.637828913Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.637891657Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.637928469Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.6379407Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.638001104Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.638158853Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.638358734Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.638481122Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.638696585Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.638916898Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.638942439Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.639007473Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.639043995Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.639327662Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.639336332Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.639343773Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.639400116Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.6394643Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.63947506Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.639540664Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.639693463Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.639878904Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.640020103Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.640258207Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.640438857Z namespaces/openshift-authentication/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.640449448Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.640460569Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.640521522Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.640705293Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.640917336Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.641062324Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.641281417Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.641515481Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.641526291Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.641532292Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.641604896Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.641688921Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.641748014Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.641815189Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.641973518Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.642160219Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.642310998Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.642533481Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.642760374Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.642775395Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.642780295Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.642823168Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.642880171Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.642888982Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.642934424Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.642993758Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.643009489Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.643060662Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.643217531Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.643413213Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.643562521Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.643929013Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.644153016Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.644194619Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.644266733Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.644312316Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.64437954Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.644409071Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.644504677Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.644599813Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.644699268Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.644754782Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.644795134Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.644828926Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.644865848Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.644965134Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.644996626Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.645050479Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.645135164Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.645176817Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.645186987Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.64524042Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.645325396Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.645372088Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.645478915Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.64556338Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.645676346Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.645793393Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.645849607Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.646031127Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.646150274Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.646169715Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.646236349Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.646284742Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.646318654Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.646380498Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.646404179Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.646489274Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.646516966Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.646618882Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.646704717Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.646772121Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.646807853Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.646861056Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.646905249Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.646942531Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.646950081Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.646994544Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.647150583Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.647333824Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.647481903Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.647703966Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.647925509Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.647957611Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.648012734Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.648020395Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.648052256Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.64811897Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.648147532Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.648219706Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.648238267Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.648333343Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.648386766Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.64844934Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.648503783Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.648591948Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.64862832Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.648812761Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.648894896Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.6489711Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.649020374Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.649064816Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.649152311Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.649191944Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.649287929Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.649339102Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.649395676Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.649508172Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.649588157Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.649597717Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.64964681Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.649756917Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.649854203Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.649970639Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.650071476Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.650120598Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.650229645Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.650308279Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.650849761Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.650936876Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.651149769Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.65134027Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.651460747Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.6514961Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.651552493Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.651593855Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.651632097Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.65167539Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.651761165Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.651786247Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.65184825Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.651923805Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.65200571Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.652044882Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.652090144Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.652138397Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.652175369Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.652246794Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.652256984Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.652320598Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.652475717Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.65269527Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.652845139Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.653074602Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.653276654Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.653288685Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.653336008Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/aws-ebs-csi-driver-node-hgb75.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.653448015Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/csi-driver/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.653457475Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/csi-driver/csi-driver/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.653463465Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/csi-driver/csi-driver/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.653502308Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.653611384Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.653697959Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.653796485Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/csi-liveness-probe/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.653806196Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.653825067Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.653865079Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.653958455Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.654026149Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.65422369Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/csi-node-driver-registrar/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.654243371Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.654251322Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.654281744Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.654379959Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.654452724Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-hgb75/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.654487856Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.654525918Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/aws-ebs-csi-driver-node-q2t2g.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.654625854Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/csi-driver/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.654633625Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/csi-driver/csi-driver/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.654646865Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/csi-driver/csi-driver/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.654713489Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.654823936Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.65489742Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.654929352Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/csi-liveness-probe/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.654936402Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.654947143Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655000456Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655092181Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655160496Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655179507Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/csi-node-driver-registrar/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655187197Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655196398Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655264972Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655354607Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655437542Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-q2t2g/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655514186Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655542308Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/aws-ebs-csi-driver-node-zl2j7.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655640174Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/csi-driver/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655655385Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/csi-driver/csi-driver/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655668986Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/csi-driver/csi-driver/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655733649Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655824395Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655890199Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655941842Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/csi-liveness-probe/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655952922Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655957803Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.655997975Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.65608753Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.656153894Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.656195077Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/csi-node-driver-registrar/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.656202237Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.656215898Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.656261981Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.656356986Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.65642682Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zl2j7/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.656459272Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.656513635Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.656584069Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.65659401Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.656647503Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.656730578Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.65676888Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.656840235Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.656858616Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.65693242Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.65694121Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.656945991Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.656988723Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.657062348Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.657081639Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.6571066Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.657158013Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.657169164Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.657239898Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.657391507Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.657572748Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.657737828Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.658050896Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.658258238Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.658305261Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.658366195Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.658406807Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.658494182Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.658509013Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.658652871Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.658758048Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.658862124Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.658910857Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.65896327Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.659041285Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.659084747Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.65913283Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.659173112Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.659258127Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.659287579Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.659340212Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.659419207Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.65946585Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.65947621Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.659526073Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.659616528Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.659661011Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.660610767Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.660742385Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.660892744Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.660975408Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.661210422Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.661422415Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.661591775Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.661750294Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.661789536Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.66185461Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.661928505Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.661984838Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.662032231Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.662095565Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.662143188Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.662232933Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.662348009Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.662405743Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.662451655Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.66252269Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.662558772Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.662636057Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.66270271Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.662853849Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.662888201Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.662954225Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.662965176Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.6630277Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.663312676Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.663510678Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.663663437Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.663952494Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.664183428Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.664259402Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-5x5qd/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.664301595Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-5x5qd/tuned-5x5qd.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.664411151Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-5x5qd/tuned/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.664419702Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-5x5qd/tuned/tuned/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.664431402Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-5x5qd/tuned/tuned/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.664476005Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-5x5qd/tuned/tuned/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.664616213Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-5x5qd/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.664704048Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-5x5qd/tuned/tuned/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.664752831Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jpg75/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.664806265Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jpg75/tuned-jpg75.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.664887669Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jpg75/tuned/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.66489618Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jpg75/tuned/tuned/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.66490079Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jpg75/tuned/tuned/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.664928452Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jpg75/tuned/tuned/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.66506224Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jpg75/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.665126333Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jpg75/tuned/tuned/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.665158055Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-m7rjb/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.665190527Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-m7rjb/tuned-m7rjb.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.665279712Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-m7rjb/tuned/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.665287713Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-m7rjb/tuned/tuned/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.665292263Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-m7rjb/tuned/tuned/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.665315255Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-m7rjb/tuned/tuned/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.665435731Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-m7rjb/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.665503355Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-m7rjb/tuned/tuned/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.665542068Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.66557943Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.665619673Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.665670125Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.66574104Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.665762021Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.665820854Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-131-45.ec2.internal.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.66591508Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-131-57.ec2.internal.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.666003885Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-142-77.ec2.internal.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.666099731Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.666141943Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.666232509Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.666285902Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.666361636Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.666403309Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.666471663Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.666504995Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.66659604Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.666728738Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.666834014Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.666880407Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.66693355Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.666978613Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.667018095Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.66710918Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.667167104Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.667202296Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.667301162Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.667376656Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.667420499Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.66761072Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.667697725Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.667859045Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.667936419Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.668061346Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.668201535Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.668412537Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.66846935Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.668497432Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.668578397Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.66862532Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.668729006Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.668773368Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.668870434Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.668949629Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.668982891Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.669027614Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.669084677Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.669115168Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.669225765Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.669253617Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.66931477Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.669354233Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.669418197Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.669425517Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.669462909Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.669618098Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.66982527Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.670080585Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.67032035Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.670514441Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.670523962Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-7vfgj/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.670560244Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-7vfgj/cluster-samples-operator-56fcf86bf-7vfgj.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.67065925Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-7vfgj/cluster-samples-operator-watch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.6706677Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-7vfgj/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.670672631Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-7vfgj/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.670797808Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-7vfgj/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.670898234Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-7vfgj/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.670964848Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-7vfgj/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.671022291Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-7vfgj/cluster-samples-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.671030181Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-7vfgj/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.671034962Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-7vfgj/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.671083975Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-7vfgj/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.67184768Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-7vfgj/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.671900903Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-7vfgj/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.671955506Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.671988408Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.672054012Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.672101555Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.67219646Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.672249324Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.672308937Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.672348149Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.672420043Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.672464126Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.672552061Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.67269617Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.672800266Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.672843608Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.672887241Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.673146936Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.673181879Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.673269544Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.673325517Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.6733743Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.673457015Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.673504287Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.673519278Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.673567101Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.673657407Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.673764803Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.673812406Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.673933473Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.674022848Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.674351768Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.674436882Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.67456314Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.674777022Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.674938242Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.675014956Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.675064659Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.675133674Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.675178856Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.67524365Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.675277522Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.675362487Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.67541192Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.675498535Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.67558233Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.675644414Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.675698567Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.675773831Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.675807543Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.675913449Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.675949332Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.676024116Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.676070449Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.676165164Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.676172725Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.676211637Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.676364076Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.676552567Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.676736408Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.676972462Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.677158123Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.677165754Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-sklbl/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.677200796Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-sklbl/volume-data-source-validator-5d456fdbcd-sklbl.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.67728362Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-sklbl/volume-data-source-validator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.677292131Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-sklbl/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.677297851Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-sklbl/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.677332883Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-sklbl/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.6774459Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-sklbl/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.677511864Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-sklbl/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.677544986Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.677573358Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.677646302Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.677654552Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.677727696Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.67778732Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.677828302Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.677883546Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.677933219Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.678008803Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.678066256Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.678133331Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.678175633Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.678246887Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.678285089Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.678370825Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.678454069Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.678536564Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.678593988Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.67864151Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.678698764Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.678857233Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.678937278Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.679001982Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.679053645Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.67914298Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.679178932Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.679187113Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.679229855Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.679325441Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.679358923Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.67947493Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.679556304Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.679658941Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.679762357Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.679841571Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.6799796Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.680174731Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.680236344Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.680287038Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.68033885Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.680382443Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.680445007Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.68048911Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.680580585Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.68066647Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.680732154Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.680777647Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.680817949Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.680873452Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.680914815Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.680940096Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.680988579Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.681142468Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.681325419Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.681468467Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.681821088Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.68201614Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.682051662Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.682118015Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.682164018Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.682265774Z namespaces/openshift-config-managed/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.682318527Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.682378051Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.682423244Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.682484067Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.68252695Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.682610444Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.68269867Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.682799276Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.682928923Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.682978436Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.68304332Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.683076372Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.683171398Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.683226021Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.683273004Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.683355248Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.683452054Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.683500207Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.68710771Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.687182814Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.68728069Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.687361175Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.687440779Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.687607749Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.687736087Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.687774119Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.687825462Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.687913917Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.687986441Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.688024654Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.688079727Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.68812522Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.688183803Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.688217055Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.688308631Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.688377895Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.688437448Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.68847592Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.688531264Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.688582217Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.68863452Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.6886451Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.688724915Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.688905816Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.689134969Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.689278918Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.689500401Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.689731855Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.689780638Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.689859762Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.689869753Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.689918115Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.689961548Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.68999854Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.690041733Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.690094736Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.690199352Z namespaces/openshift-config-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.690209273Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.690214893Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.690250195Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.690310949Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.690321189Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.690375753Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.690530192Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.690726193Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.690868322Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.691058093Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.691274175Z namespaces/openshift-config/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.691322338Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.691502249Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.691544771Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.691627086Z namespaces/openshift-config/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.691670399Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.691777195Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.69185792Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.691934674Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.691972697Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.692018629Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.692054892Z namespaces/openshift-config/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.692106495Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.69218992Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.692221151Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.692270864Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.692349539Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.692397962Z namespaces/openshift-config/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.692440664Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.692557661Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.692662257Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.692764903Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.692840478Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.692918483Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.693073302Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.69321784Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.693276554Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.693323406Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.69338015Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.693425402Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.693472295Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.693514758Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.693568111Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.693615564Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.693691908Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.693800555Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.693852687Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.69390009Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.693935383Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.693983235Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.694041599Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.694049869Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.694096292Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.69424259Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.694418301Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.694552309Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.694878578Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.695059039Z namespaces/openshift-config/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.695100081Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.695145584Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.695174645Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.695280982Z namespaces/openshift-console-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.695318614Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.695481474Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.695515576Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.695575749Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.695604191Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.695696516Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.695839335Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.695963442Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.696011005Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.696069478Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.696119711Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.696183825Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.69626832Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.696314853Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.696355555Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.69643945Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.696487343Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.696496383Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.696529965Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.6966048Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.696656483Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.697537195Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.69762566Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.697904726Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.697989162Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.698112639Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.698248827Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.698453559Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.698522403Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.698580926Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.69863204Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.698693793Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.698752276Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.698799219Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.698882714Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.698965199Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.699024793Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.699080686Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.69915415Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.699172021Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.699240115Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.699276498Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.699333011Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.699360883Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.699415976Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.699425306Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.69948183Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.699631598Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.69983528Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.699972779Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.700185391Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.700376292Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.700386293Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-pn8bl/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.700434596Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-pn8bl/console-operator-68b757fb67-pn8bl.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.700625197Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-pn8bl/console-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.700636048Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-pn8bl/console-operator/console-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.700643088Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-pn8bl/console-operator/console-operator/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.700691441Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-pn8bl/console-operator/console-operator/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.701207051Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-pn8bl/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.701260524Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.701306347Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.701361661Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.701416854Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.701477427Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.701487918Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.701496298Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.70152186Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.701707561Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.701899282Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.70203355Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.702275724Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.702477516Z namespaces/openshift-console/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.702520059Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.702671308Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.702795155Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.702848478Z namespaces/openshift-console/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.702891481Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.702977966Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.703139085Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.703430852Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.703484376Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.703540879Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.703588572Z namespaces/openshift-console/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.703632574Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.703739581Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.703800334Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.703854248Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.703925862Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.703971594Z namespaces/openshift-console/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.704019627Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.70474194Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.704832535Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.705138373Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.705219218Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.705393388Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.705517895Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.705710487Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.705791292Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.705822993Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.705901418Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.705950201Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.706009044Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.706054177Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.706142112Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.706256149Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.706312733Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.706363885Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.706426289Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.706478912Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.706533515Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.706590389Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.706840804Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.706851484Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.706897777Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.707047406Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.707231907Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.707373095Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.707566807Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.70779932Z namespaces/openshift-console/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.707809671Z namespaces/openshift-console/pods/console-69f57c9ff6-b62bc/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.707848163Z namespaces/openshift-console/pods/console-69f57c9ff6-b62bc/console-69f57c9ff6-b62bc.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.707942339Z namespaces/openshift-console/pods/console-69f57c9ff6-b62bc/console/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.707953439Z namespaces/openshift-console/pods/console-69f57c9ff6-b62bc/console/console/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.70795841Z namespaces/openshift-console/pods/console-69f57c9ff6-b62bc/console/console/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.707987271Z namespaces/openshift-console/pods/console-69f57c9ff6-b62bc/console/console/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.708087627Z namespaces/openshift-console/pods/console-69f57c9ff6-b62bc/console/console/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.708154341Z namespaces/openshift-console/pods/console-69f57c9ff6-b62bc/console/console/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.708180273Z namespaces/openshift-console/pods/downloads-685bfb5f64-qkh89/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.708228846Z namespaces/openshift-console/pods/downloads-685bfb5f64-qkh89/downloads-685bfb5f64-qkh89.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.708316261Z namespaces/openshift-console/pods/downloads-685bfb5f64-qkh89/download-server/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.708323411Z namespaces/openshift-console/pods/downloads-685bfb5f64-qkh89/download-server/download-server/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.708327381Z namespaces/openshift-console/pods/downloads-685bfb5f64-qkh89/download-server/download-server/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.708359033Z namespaces/openshift-console/pods/downloads-685bfb5f64-qkh89/download-server/download-server/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.70880953Z namespaces/openshift-console/pods/downloads-685bfb5f64-qkh89/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.708876234Z namespaces/openshift-console/pods/downloads-685bfb5f64-qkh89/download-server/download-server/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.708901535Z namespaces/openshift-console/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.708950068Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.709030993Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.709065215Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.709179562Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.709219384Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.709281748Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.709316179Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.709374503Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.709403145Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.709552113Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.709637758Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.709745215Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.709805198Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.709850881Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.709906945Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.709952337Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.710029112Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.710086365Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.710141708Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.710213503Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.710281527Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71033766Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.710446416Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.710520451Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.710600605Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71067584Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.710860911Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.710979208Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.711134727Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71119374Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.711249314Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.711303347Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.711346919Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.711395432Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.711446955Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71152719Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.711608255Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.711652908Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71169461Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.711769904Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.711819257Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.711855159Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71186148Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.711905052Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.712055211Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.712235732Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71236875Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.712565241Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.712790595Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.712809446Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.712871879Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.712904831Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.713009207Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71305137Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.713121624Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.713157456Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71321912Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.713262542Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.713344247Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.713425162Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.713504407Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71356328Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.713610803Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.713655736Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71372607Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.713811985Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.713870488Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.713918371Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.713995136Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.714128303Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.714159135Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.714214868Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.714295703Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.714340706Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.714449772Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.714524217Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.714610802Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.714694717Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.714801303Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71492059Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71508268Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.715172235Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.715214158Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.715269551Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.715309143Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.715351386Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.715404799Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.715485363Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.715562428Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.715608031Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.715649673Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.715694566Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71575781Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.715799232Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.715808462Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.715840444Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.715995553Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.716171644Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.716317143Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.716515794Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.716887426Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.716930569Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.716992332Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.717020654Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.717104499Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.717151122Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.717209015Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.717249007Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.717318412Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.717369945Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71745451Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.717533624Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.717612759Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.717666632Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.717724595Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.717788719Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.717815521Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.717915517Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71797308Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.718012032Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.718094727Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.718170742Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.718219625Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.718323921Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.718399785Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.718491321Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.718572085Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71865493Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.718884824Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.719051524Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.719107197Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.719146379Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.719198802Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.719249325Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.719280297Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71932581Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.719412315Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71949096Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.719540153Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.719566064Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.719626298Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71966827Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.719757666Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.719778097Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.719841591Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.719873812Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.719929716Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.719939076Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.71999898Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.720150989Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.72033941Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.720478848Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.720704341Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.720885412Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.720919804Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.720958226Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.721009119Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.721092834Z namespaces/openshift-dns/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.721135787Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.721211251Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.721242003Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.722052291Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.722096353Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.722244472Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.722321877Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.722411492Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.722467105Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.722516458Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.722575681Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.722628445Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.722735361Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.722790664Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.722824306Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.722916742Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.722984386Z namespaces/openshift-dns/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.723039489Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.723129714Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.723214879Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.723433882Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.723510717Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.723831816Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.723950483Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.724132124Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.724204928Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.724254401Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.724318444Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.724361687Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.724439962Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.724473293Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.724571299Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.724663535Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.724729519Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.724779882Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.724835775Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.724874877Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.72493146Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.725029746Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.725084269Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.7250928Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.725132022Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.725310543Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.725506825Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.725644533Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.725871236Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.726086679Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.726094229Z namespaces/openshift-dns/pods/dns-default-2p2vp/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.726141482Z namespaces/openshift-dns/pods/dns-default-2p2vp/dns-default-2p2vp.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.726233837Z namespaces/openshift-dns/pods/dns-default-2p2vp/dns/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.726241288Z namespaces/openshift-dns/pods/dns-default-2p2vp/dns/dns/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.726249568Z namespaces/openshift-dns/pods/dns-default-2p2vp/dns/dns/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.726333593Z namespaces/openshift-dns/pods/dns-default-2p2vp/dns/dns/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.72644272Z namespaces/openshift-dns/pods/dns-default-2p2vp/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.726511774Z namespaces/openshift-dns/pods/dns-default-2p2vp/dns/dns/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.726542046Z namespaces/openshift-dns/pods/dns-default-2p2vp/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.726557167Z namespaces/openshift-dns/pods/dns-default-2p2vp/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.726561847Z namespaces/openshift-dns/pods/dns-default-2p2vp/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.72661397Z namespaces/openshift-dns/pods/dns-default-2p2vp/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.726818172Z namespaces/openshift-dns/pods/dns-default-2p2vp/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.726887336Z namespaces/openshift-dns/pods/dns-default-2p2vp/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.726915017Z namespaces/openshift-dns/pods/dns-default-nb4m7/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.726970711Z namespaces/openshift-dns/pods/dns-default-nb4m7/dns-default-nb4m7.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.727054376Z namespaces/openshift-dns/pods/dns-default-nb4m7/dns/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.727061986Z namespaces/openshift-dns/pods/dns-default-nb4m7/dns/dns/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.727069087Z namespaces/openshift-dns/pods/dns-default-nb4m7/dns/dns/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.727159602Z namespaces/openshift-dns/pods/dns-default-nb4m7/dns/dns/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.727251487Z namespaces/openshift-dns/pods/dns-default-nb4m7/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.727325942Z namespaces/openshift-dns/pods/dns-default-nb4m7/dns/dns/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.727359794Z namespaces/openshift-dns/pods/dns-default-nb4m7/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.727379315Z namespaces/openshift-dns/pods/dns-default-nb4m7/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.727395636Z namespaces/openshift-dns/pods/dns-default-nb4m7/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.727452459Z namespaces/openshift-dns/pods/dns-default-nb4m7/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.727546615Z namespaces/openshift-dns/pods/dns-default-nb4m7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.727609489Z namespaces/openshift-dns/pods/dns-default-nb4m7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.72763846Z namespaces/openshift-dns/pods/dns-default-rps8j/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.727695783Z namespaces/openshift-dns/pods/dns-default-rps8j/dns-default-rps8j.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.727903346Z namespaces/openshift-dns/pods/dns-default-rps8j/dns/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.727911396Z namespaces/openshift-dns/pods/dns-default-rps8j/dns/dns/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.727918367Z namespaces/openshift-dns/pods/dns-default-rps8j/dns/dns/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.72797334Z namespaces/openshift-dns/pods/dns-default-rps8j/dns/dns/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.728085177Z namespaces/openshift-dns/pods/dns-default-rps8j/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.728156281Z namespaces/openshift-dns/pods/dns-default-rps8j/dns/dns/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.728193043Z namespaces/openshift-dns/pods/dns-default-rps8j/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.728202453Z namespaces/openshift-dns/pods/dns-default-rps8j/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.728206984Z namespaces/openshift-dns/pods/dns-default-rps8j/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.728269917Z namespaces/openshift-dns/pods/dns-default-rps8j/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.728369883Z namespaces/openshift-dns/pods/dns-default-rps8j/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.728439827Z namespaces/openshift-dns/pods/dns-default-rps8j/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.7284856Z namespaces/openshift-dns/pods/node-resolver-228pr/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.728529873Z namespaces/openshift-dns/pods/node-resolver-228pr/node-resolver-228pr.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.728639149Z namespaces/openshift-dns/pods/node-resolver-228pr/dns-node-resolver/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.72865149Z namespaces/openshift-dns/pods/node-resolver-228pr/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.72865697Z namespaces/openshift-dns/pods/node-resolver-228pr/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.728696782Z namespaces/openshift-dns/pods/node-resolver-228pr/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.728829861Z namespaces/openshift-dns/pods/node-resolver-228pr/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.728898834Z namespaces/openshift-dns/pods/node-resolver-228pr/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.728962798Z namespaces/openshift-dns/pods/node-resolver-fp74t/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.72899113Z namespaces/openshift-dns/pods/node-resolver-fp74t/node-resolver-fp74t.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729076115Z namespaces/openshift-dns/pods/node-resolver-fp74t/dns-node-resolver/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729083485Z namespaces/openshift-dns/pods/node-resolver-fp74t/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729088206Z namespaces/openshift-dns/pods/node-resolver-fp74t/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729120988Z namespaces/openshift-dns/pods/node-resolver-fp74t/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729213063Z namespaces/openshift-dns/pods/node-resolver-fp74t/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729285897Z namespaces/openshift-dns/pods/node-resolver-fp74t/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729314219Z namespaces/openshift-dns/pods/node-resolver-hx4xq/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729352481Z namespaces/openshift-dns/pods/node-resolver-hx4xq/node-resolver-hx4xq.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729433226Z namespaces/openshift-dns/pods/node-resolver-hx4xq/dns-node-resolver/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729441357Z namespaces/openshift-dns/pods/node-resolver-hx4xq/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729445977Z namespaces/openshift-dns/pods/node-resolver-hx4xq/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729480319Z namespaces/openshift-dns/pods/node-resolver-hx4xq/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729576845Z namespaces/openshift-dns/pods/node-resolver-hx4xq/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729640398Z namespaces/openshift-dns/pods/node-resolver-hx4xq/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729704912Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729723443Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729787517Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729820919Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729922315Z namespaces/openshift-etcd/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.729966548Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.730024491Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.730053853Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.730113306Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.730143038Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.730241444Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.730320658Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.730419354Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.730459297Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.730494979Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.730555402Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.730598835Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.7306961Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.730764435Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.730809647Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.730897483Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.730968287Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.73102139Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.731119576Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.73119525Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.731278005Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.73136022Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.731439144Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.731560911Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.731752553Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.731801816Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.731846978Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.731891991Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.731939364Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.731996077Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.73204451Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.732126155Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.73221153Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.732244152Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.732300475Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.732339688Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.7323883Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.732423682Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.732461735Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.732508218Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.732689728Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.732880389Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.733030348Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.73322565Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.733431082Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.733476764Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.733518847Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.73356452Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.733658905Z namespaces/openshift-host-network/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.733730659Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.733806564Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.733852907Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.733923281Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.733968764Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.734060669Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.734143664Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.73424342Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.734261781Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.734306024Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.734360437Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.734403429Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.734491674Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.734554258Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.734606691Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.734691126Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.734775311Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.734812633Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.73492456Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.735004715Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.735086879Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.735181195Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.735275571Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.735422779Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.735527316Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.735592139Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.735639112Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.735696815Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.735859445Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.735908058Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.735965401Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.736009534Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.736050366Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.736140522Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.736220216Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.736269489Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.736318622Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.736356845Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.736401457Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.73645737Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.736468551Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.736524574Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.736704665Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.736987122Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.73713328Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.737343943Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.737551425Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.737580806Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.73764056Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.737692363Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.737789699Z namespaces/openshift-image-registry/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.737837072Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.737904876Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.737937968Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.737990581Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.738022493Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.73813871Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.738280068Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.7384795Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.738508951Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.738560284Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.738611858Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.738665671Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.738807529Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.73883166Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.738897804Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.738976119Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.739013301Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.739023732Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.739084065Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.739150809Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.739209653Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.740242923Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.740390922Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.740796126Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.740881941Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.741103694Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.741230382Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.741424043Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.741483667Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.74154203Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.741574542Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.741630056Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.741694429Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.741768404Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.741860929Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.741942294Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.742005588Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.742090853Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.742159577Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.742202019Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.742330987Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.742420042Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.742470725Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.742528118Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.74256047Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.742611423Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.742658406Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.742672467Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.742765972Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.742937993Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.743126034Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.743282443Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.743511226Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.743830105Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.743842426Z namespaces/openshift-image-registry/pods/image-registry-55cdb97ff5-g8zn2/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.74390584Z namespaces/openshift-image-registry/pods/image-registry-55cdb97ff5-g8zn2/image-registry-55cdb97ff5-g8zn2.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.744031177Z namespaces/openshift-image-registry/pods/image-registry-55cdb97ff5-g8zn2/registry/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.744047278Z namespaces/openshift-image-registry/pods/image-registry-55cdb97ff5-g8zn2/registry/registry/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.744052878Z namespaces/openshift-image-registry/pods/image-registry-55cdb97ff5-g8zn2/registry/registry/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.744096821Z namespaces/openshift-image-registry/pods/image-registry-55cdb97ff5-g8zn2/registry/registry/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.746650601Z namespaces/openshift-image-registry/pods/image-registry-55cdb97ff5-g8zn2/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.746746147Z namespaces/openshift-image-registry/pods/image-registry-55cdb97ff5-g8zn2/registry/registry/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.746770148Z namespaces/openshift-image-registry/pods/node-ca-4jcp4/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.746827612Z namespaces/openshift-image-registry/pods/node-ca-4jcp4/node-ca-4jcp4.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.746906807Z namespaces/openshift-image-registry/pods/node-ca-4jcp4/node-ca/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.746916837Z namespaces/openshift-image-registry/pods/node-ca-4jcp4/node-ca/node-ca/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.746925468Z namespaces/openshift-image-registry/pods/node-ca-4jcp4/node-ca/node-ca/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.746938729Z namespaces/openshift-image-registry/pods/node-ca-4jcp4/node-ca/node-ca/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.747160582Z namespaces/openshift-image-registry/pods/node-ca-4jcp4/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.747231586Z namespaces/openshift-image-registry/pods/node-ca-4jcp4/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.747268928Z namespaces/openshift-image-registry/pods/node-ca-9627c/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.747327392Z namespaces/openshift-image-registry/pods/node-ca-9627c/node-ca-9627c.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.747412237Z namespaces/openshift-image-registry/pods/node-ca-9627c/node-ca/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.747423257Z namespaces/openshift-image-registry/pods/node-ca-9627c/node-ca/node-ca/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.747429027Z namespaces/openshift-image-registry/pods/node-ca-9627c/node-ca/node-ca/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.74746985Z namespaces/openshift-image-registry/pods/node-ca-9627c/node-ca/node-ca/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.747655281Z namespaces/openshift-image-registry/pods/node-ca-9627c/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.747756947Z namespaces/openshift-image-registry/pods/node-ca-9627c/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.747789289Z namespaces/openshift-image-registry/pods/node-ca-mqszb/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.747838142Z namespaces/openshift-image-registry/pods/node-ca-mqszb/node-ca-mqszb.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.747914866Z namespaces/openshift-image-registry/pods/node-ca-mqszb/node-ca/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.747927657Z namespaces/openshift-image-registry/pods/node-ca-mqszb/node-ca/node-ca/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.747932447Z namespaces/openshift-image-registry/pods/node-ca-mqszb/node-ca/node-ca/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.74798472Z namespaces/openshift-image-registry/pods/node-ca-mqszb/node-ca/node-ca/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.748181512Z namespaces/openshift-image-registry/pods/node-ca-mqszb/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.748252126Z namespaces/openshift-image-registry/pods/node-ca-mqszb/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.748279368Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.748331701Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.748397335Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.748428766Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.748478989Z namespaces/openshift-infra/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.74848829Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.7484929Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.748529592Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.748734524Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.748919545Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.749057514Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.749255675Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.749496399Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.749528781Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.749618587Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.749650998Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.749742554Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.749781116Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.749905393Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.749977778Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.750064993Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.750118266Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.750170959Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.750201861Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.750263095Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.75034631Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.750402773Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.750460236Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.750542881Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.750630146Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.750664318Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.750817437Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.750904483Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.751055801Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.751133316Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.751298816Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.751423303Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.751601304Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.751669737Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.751744692Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.751816726Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.751857349Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.751913412Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.751963395Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.7520521Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.752135295Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.752192208Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.752241031Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.752307045Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.752364669Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.752410491Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.752423392Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.752478295Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.752631774Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.752871728Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.753006776Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.753223169Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.753504976Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.753514777Z namespaces/openshift-ingress-canary/pods/ingress-canary-57cbl/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.753561789Z namespaces/openshift-ingress-canary/pods/ingress-canary-57cbl/ingress-canary-57cbl.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.753647194Z namespaces/openshift-ingress-canary/pods/ingress-canary-57cbl/serve-healthcheck-canary/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.753655765Z namespaces/openshift-ingress-canary/pods/ingress-canary-57cbl/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.753668006Z namespaces/openshift-ingress-canary/pods/ingress-canary-57cbl/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.75374056Z namespaces/openshift-ingress-canary/pods/ingress-canary-57cbl/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.75391397Z namespaces/openshift-ingress-canary/pods/ingress-canary-57cbl/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.753981314Z namespaces/openshift-ingress-canary/pods/ingress-canary-57cbl/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.754030637Z namespaces/openshift-ingress-canary/pods/ingress-canary-lk74h/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.75407994Z namespaces/openshift-ingress-canary/pods/ingress-canary-lk74h/ingress-canary-lk74h.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.754149784Z namespaces/openshift-ingress-canary/pods/ingress-canary-lk74h/serve-healthcheck-canary/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.754157694Z namespaces/openshift-ingress-canary/pods/ingress-canary-lk74h/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.754188886Z namespaces/openshift-ingress-canary/pods/ingress-canary-lk74h/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.754232579Z namespaces/openshift-ingress-canary/pods/ingress-canary-lk74h/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.754330165Z namespaces/openshift-ingress-canary/pods/ingress-canary-lk74h/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.754399009Z namespaces/openshift-ingress-canary/pods/ingress-canary-lk74h/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.754605561Z namespaces/openshift-ingress-canary/pods/ingress-canary-vr2td/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.754649283Z namespaces/openshift-ingress-canary/pods/ingress-canary-vr2td/ingress-canary-vr2td.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.75476354Z namespaces/openshift-ingress-canary/pods/ingress-canary-vr2td/serve-healthcheck-canary/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.754774811Z namespaces/openshift-ingress-canary/pods/ingress-canary-vr2td/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.754785401Z namespaces/openshift-ingress-canary/pods/ingress-canary-vr2td/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.754829974Z namespaces/openshift-ingress-canary/pods/ingress-canary-vr2td/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.75493007Z namespaces/openshift-ingress-canary/pods/ingress-canary-vr2td/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.754998844Z namespaces/openshift-ingress-canary/pods/ingress-canary-vr2td/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.755038846Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.755076509Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.755133462Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.755183115Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.755293871Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.755342074Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.755417959Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.755461781Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.755526555Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.755561857Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.755652603Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.755763189Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.755849094Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.755910498Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.755976162Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.755995023Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.756052666Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.756131451Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.756193934Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.756238407Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.756323002Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.756386966Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.756441089Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.757370794Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.757465999Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.757590817Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.757696263Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.75780603Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.757927117Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.758114228Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.758183652Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.758242425Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.758288738Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.758346291Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.758397544Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.758406815Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.758451517Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.758530242Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.758569824Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.75866456Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.758794308Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.758854211Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.758914865Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.758962108Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.759027971Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.759109756Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.759144948Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.759210312Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.759253875Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.759302908Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.759316459Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.759354231Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.759449866Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.759465427Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.75951098Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.759670239Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.759911904Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.760053422Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.760279535Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.760614395Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.760666968Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.760728942Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.760774274Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.760896912Z namespaces/openshift-ingress/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.760937814Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.761130006Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.761177128Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.761253843Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.761286885Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.76138117Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.761619484Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.761858878Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.761906701Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.761960775Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.762028899Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.762062021Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.762156756Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.762211079Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.762262192Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.762352338Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.762385129Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.76239638Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.762453933Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.762552449Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.762636494Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.762767062Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.762814895Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.763256801Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.763364357Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.763587411Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.763669575Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.763955312Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.764075519Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.764307483Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.764395788Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.76442496Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.764514355Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.764524736Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.764580489Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.764672804Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.764734658Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.764820103Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.764871346Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.764949321Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.765034826Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.765097009Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.765150023Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.765212896Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.765262149Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.765312932Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.765358915Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.765392737Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.765431469Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.765475662Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.765729617Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.76594953Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.766093418Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.766320312Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.766528584Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.766536754Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.766577657Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/istiod-openshift-gateway-94bb8fbfd-zmx9k.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.766698524Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/discovery/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.766713245Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/discovery/discovery/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.766721975Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/discovery/discovery/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.766787439Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/discovery/discovery/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.789943235Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.790015159Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-zmx9k/discovery/discovery/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.790078973Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.790128036Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/openshift-ai-inference-openshift-default-9455cf8d5-vmnww.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.790260624Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/istio-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.790496658Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/istio-proxy/istio-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.790508758Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.790555871Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.790851458Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-9455cf8d5-vmnww/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.790946584Z namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.790981526Z namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router-default-757d9b65f5-7xf7w.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.791101733Z namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.791110614Z namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router/router/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.791115674Z namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router/router/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.791165007Z namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router/router/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.791275153Z namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router/router/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.791343097Z namespaces/openshift-ingress/pods/router-default-757d9b65f5-7xf7w/router/router/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.791406101Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.791417822Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.791490806Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.791542079Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.791644535Z namespaces/openshift-insights/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.79172339Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.791795764Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.791852638Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.791910051Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.791950263Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.792080561Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.792219229Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.792325115Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.792368268Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.79240595Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.792485585Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.792534428Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.792605722Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.792647284Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.792759321Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.792801934Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.792857267Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.792936171Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.793012006Z namespaces/openshift-insights/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.793068699Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.793954471Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.794067498Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.794413169Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.794547917Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.794898147Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.794985402Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.795144692Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.795209525Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.795264649Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.795324972Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.795378786Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.795463031Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.795521484Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.795576817Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.795633861Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.795695204Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.795766368Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.79580106Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.795865964Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.795914427Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.795999542Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.796079377Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.79613776Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.796166582Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.796251207Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.796290039Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.796371074Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.796411716Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.79647568Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.796523993Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.796581706Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.796593517Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.79664717Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.796868783Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.797076746Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.797231085Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.797441067Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.797695292Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.797708953Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-kjjpn/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.797741115Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-kjjpn/insights-operator-5c847dcfc4-kjjpn.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.797846091Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-kjjpn/insights-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.797854681Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-kjjpn/insights-operator/insights-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.797862442Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-kjjpn/insights-operator/insights-operator/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.797909705Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-kjjpn/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.798285097Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-kjjpn/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.798629907Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.798657639Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/insights-runtime-extractor-4ct7m.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.798781856Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/exporter/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.798793427Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/exporter/exporter/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.798797407Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/exporter/exporter/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.798821589Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/exporter/exporter/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.798925555Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.798993689Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/exporter/exporter/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.799023841Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/extractor/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.799033091Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/extractor/extractor/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.799036981Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/extractor/extractor/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.799077693Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/extractor/extractor/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.79918183Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.799247483Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/extractor/extractor/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.799284906Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.799302487Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.799308237Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.79935841Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.799455056Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.79952001Z namespaces/openshift-insights/pods/insights-runtime-extractor-4ct7m/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.799571773Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.799612615Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/insights-runtime-extractor-cmkcf.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.799749253Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/exporter/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.799760584Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/exporter/exporter/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.799770274Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/exporter/exporter/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.799820377Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/exporter/exporter/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.799912053Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.800003358Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/exporter/exporter/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.800015139Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/extractor/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.800021399Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/extractor/extractor/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.80002536Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/extractor/extractor/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.800079523Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/extractor/extractor/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.800173828Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.800250963Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/extractor/extractor/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.800271184Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.800277144Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.800284345Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.800321777Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.800413112Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.800483836Z namespaces/openshift-insights/pods/insights-runtime-extractor-cmkcf/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.800704049Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.800744982Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/insights-runtime-extractor-fl8wb.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.800837337Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/exporter/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.800846858Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/exporter/exporter/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.800852558Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/exporter/exporter/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.800892411Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/exporter/exporter/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.800986736Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.80105532Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/exporter/exporter/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801089172Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/extractor/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801096283Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/extractor/extractor/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801100133Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/extractor/extractor/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801148846Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/extractor/extractor/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801241681Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801312605Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/extractor/extractor/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801338767Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801346098Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801354898Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801401661Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801488846Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.80156097Z namespaces/openshift-insights/pods/insights-runtime-extractor-fl8wb/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801595522Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801624654Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801701978Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.80173245Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801794434Z namespaces/openshift-keda/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801803254Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801807254Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801851207Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801942873Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801990976Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.801999156Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.802038798Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.802088761Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.802130244Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.802225339Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.802312484Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.802371798Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.802381739Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.802422711Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.802565549Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.802773331Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.802963813Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.803192956Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.803338405Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.803502705Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.803545577Z namespaces/openshift-keda/operators.coreos.com/installplans/install-b85dd.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.803656304Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.803716737Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.803777151Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.803810883Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.803869226Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.803902468Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.804000954Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.804034856Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.80410644Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.804143272Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.804204366Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.804231628Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.804324593Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.804400707Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.804484923Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.804533305Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.804588559Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.804659043Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.804717366Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.804812032Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.804856325Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.804900517Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.804980702Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.805044936Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.805098019Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.805203355Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.805282759Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.805363534Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.805444089Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.805524784Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.805663612Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.805856303Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.805894086Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.805943678Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.805990951Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.806038914Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.806088017Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.806130129Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.806210044Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.806289679Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.806419187Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.806460929Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.806514242Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.806551414Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.806607008Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.806614008Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.80665414Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.806829901Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.807006801Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.807163101Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.807372583Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.807560624Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.807591516Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.807653429Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.807670551Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.807820929Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.807868722Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.807930836Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.807968648Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.808043512Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.808076864Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.808164749Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.808243694Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.808326069Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.808380272Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.808411524Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.808484438Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.808535671Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.808625877Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.808660199Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.808735173Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.808817308Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.808876022Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.808919474Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.80902569Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.809105715Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.80918885Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.809269165Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.80935287Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.809471477Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.809633706Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.8096907Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.809759194Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.809813107Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.80986388Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.809917913Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.809966476Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.810045851Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.810128155Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.81020331Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.810242752Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.810304936Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.810344348Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.810444974Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.810511568Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.810558361Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.810621494Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.810670047Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.810743972Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.810760763Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.810811676Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.810967705Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.811140875Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.811283253Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.811510897Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.811719029Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.811754221Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.811824885Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.811867238Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.811966764Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.812014967Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.812089641Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.812119823Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.812200127Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.812232169Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.812325215Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.812400289Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.812484754Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.812518306Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.81258259Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.812641984Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.812692007Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.812800503Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.812851666Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.812898839Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.812975193Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.813056458Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.813097331Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.813209207Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.813288772Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.813370507Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.813450481Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.813533496Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.813657024Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.813852655Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.813921039Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.813972202Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.814041166Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.814086649Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.814141262Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.814194175Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.814265019Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.814351814Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.814386087Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.814435859Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.814492093Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.814545196Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.814585828Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.814634771Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.814700045Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.814711626Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.814755818Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.814912427Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.815088268Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.815228056Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.815420588Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.815605799Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.815644501Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.815710845Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.81580659Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.815866244Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.815919577Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.81597214Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.816010432Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.816061695Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.816119919Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.816191113Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.816269058Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.816357593Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.816402615Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.816449488Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.816506292Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.816559495Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.81664649Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.816697563Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.816761387Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.816841641Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.816896475Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.816903985Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.816940817Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.817008261Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.817040023Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.817148359Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.817229024Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.817508231Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.817587845Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.81766739Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.817824009Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.81800928Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.818059273Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.818109616Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.818151959Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.818195511Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.818254475Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.818307528Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.818378922Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.818461207Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.818493059Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.818548012Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.818596395Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.818640058Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.81869099Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.818701421Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.818749924Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.818902113Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.81919561Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.819331618Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.81952663Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.819729212Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.819772054Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.819819117Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.8198664Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.819931503Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.819979136Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.82004774Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.820094673Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.820151616Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.820193149Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.820277664Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.820355268Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.820435573Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.820484646Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.820528389Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.820585112Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.820630065Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.820733181Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.820778704Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.820826656Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.820904921Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.820958194Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.821008877Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.821123214Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.821206249Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.821287494Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.821372568Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.821470784Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.821632294Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.821852367Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.821913031Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.821964903Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.822007156Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.822047968Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.822098822Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.822143964Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.822223649Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.822305063Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.822352686Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.822402629Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.822445162Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.822492805Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.822534507Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.822541557Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.822595271Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.82275535Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.822935711Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.823070489Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.823273921Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.823469632Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.823501534Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.823571958Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.82360363Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.823702526Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.823840134Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.823894497Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.82394708Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.823986593Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.824037646Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.824121141Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.824199345Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.82427893Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.824317752Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.824364595Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.824419508Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.824464541Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.824542855Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.824592158Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.824644461Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.824750688Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.824803331Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.824831152Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.824940159Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.825016543Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.825097898Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.825173853Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.8252911Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.825426248Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.8256398Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.825673092Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.825759867Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.825793349Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.825845282Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.825877384Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.825942658Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.826044154Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.826110748Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.826153521Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.826233095Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.826280308Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.82631568Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.826371433Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.826378574Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.826411496Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.826574145Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.826782638Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.826931276Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.827142839Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.82733175Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.827372422Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.827428166Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.827445957Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.827541712Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.827579565Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.827644948Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.827695651Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.827760645Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.827798847Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.827893083Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.82801592Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.828119066Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.828170689Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.828220992Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.828273355Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.828327129Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.828409583Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.828456626Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.828505109Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.828602065Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.828635347Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.828646128Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.828730553Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.828819708Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.82886333Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.829000738Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.829096504Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.829275735Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.82935697Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.829489047Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.829636566Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.829848048Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.829922253Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.829998217Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.830060771Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.830111644Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.830180288Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.83021827Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.830304915Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.83038754Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.830445954Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.830498707Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.83055317Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.830604633Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.830660576Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.830670177Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.830753612Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.830903121Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.831084611Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.831223889Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.831428332Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.831801614Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.831812964Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-tmvn6/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.831866278Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-tmvn6/kube-storage-version-migrator-operator-846bf668f7-tmvn6.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.832003746Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-tmvn6/kube-storage-version-migrator-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.832017366Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-tmvn6/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.832044988Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-tmvn6/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.832111432Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-tmvn6/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.832236769Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-tmvn6/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.832356666Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.832396849Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.832450262Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.832492454Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.832547568Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.832607661Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.832667965Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.832722888Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.832793712Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.832825664Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.832953831Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.833061268Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.833161094Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.833216857Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.833254169Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.833304802Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.833337724Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.833419689Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.833467672Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.833518255Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.83359798Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.833637152Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.833705276Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.833816842Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.833888627Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.834015944Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.834096769Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.834209556Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.834336413Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.834501283Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.834554916Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.834585408Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.834649092Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.834704905Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.834777369Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.834808491Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.834910797Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.834988162Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.835026874Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.835061946Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.835119719Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.835164532Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.835217475Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.835225045Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.835267788Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.835416127Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.835595367Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.835755417Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.835946698Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.836128189Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.836136809Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-97pd6/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.836177502Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-97pd6/migrator-d57d58cc9-97pd6.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.836294288Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-97pd6/graceful-termination/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.836302819Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-97pd6/graceful-termination/graceful-termination/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.83631434Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-97pd6/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.836357242Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-97pd6/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.836461469Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-97pd6/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.836526792Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-97pd6/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.836562524Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-97pd6/migrator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.836577025Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-97pd6/migrator/migrator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.836582176Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-97pd6/migrator/migrator/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.836633328Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-97pd6/migrator/migrator/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.836753686Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-97pd6/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.83683364Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-97pd6/migrator/migrator/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.836849851Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.836912745Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.836948957Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.837059054Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.837118627Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.8371627Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.837217053Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.837257725Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.837322129Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.837370332Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.837446416Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.837600786Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.837745474Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.837804878Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.837865691Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.837887933Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.837943666Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.838031531Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.838068123Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.838121166Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.838208921Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.838249174Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.838255914Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.838304457Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.838389742Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.838456596Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.838508029Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.838613445Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.838734272Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.839131396Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.839212811Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.839398102Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.839523159Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.839820716Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.839887411Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.839922262Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.839990706Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.840026439Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.840084052Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.840116364Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.840209239Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.840287664Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.840344378Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.840406321Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.840453914Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.840513008Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.84056048Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.840619204Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.840664977Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.840672727Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.840749642Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.840906581Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.841083841Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.841225659Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.841462864Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.841708918Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.841898749Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.841933121Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-zcgkr.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.842022986Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.842070229Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.842122792Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.842167195Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.842225909Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.84225735Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.842335215Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.842343875Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7845fd6587-tfltn/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.842384328Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7845fd6587-tfltn/lws-controller-manager-7845fd6587-tfltn.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.842474873Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7845fd6587-tfltn/manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.842488434Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7845fd6587-tfltn/manager/manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.842495204Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7845fd6587-tfltn/manager/manager/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.842533856Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7845fd6587-tfltn/manager/manager/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.842789862Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7845fd6587-tfltn/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.842856175Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7845fd6587-tfltn/manager/manager/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.842907829Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7845fd6587-tgmtj/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.842940531Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7845fd6587-tgmtj/lws-controller-manager-7845fd6587-tgmtj.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843026286Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7845fd6587-tgmtj/manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843034186Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7845fd6587-tgmtj/manager/manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843038896Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7845fd6587-tgmtj/manager/manager/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843065998Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7845fd6587-tgmtj/manager/manager/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843185005Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7845fd6587-tgmtj/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843255079Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7845fd6587-tgmtj/manager/manager/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843290781Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-r25kc/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843327534Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-r25kc/openshift-lws-operator-fd8ccff4c-r25kc.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843413779Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-r25kc/openshift-lws-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843421189Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-r25kc/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843428329Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-r25kc/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843476952Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-r25kc/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843650122Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-r25kc/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843739868Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-r25kc/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843770049Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843818012Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843866695Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843916668Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843961571Z namespaces/openshift-machine-api/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843969021Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.843973492Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.844012024Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.844247548Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.844429968Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.844574057Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.84479814Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.844980451Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.844988141Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.844992722Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.845025574Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.8451272Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.845212345Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.845219825Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.845267008Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.845418427Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.845615748Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.845783328Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.846016112Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.846225174Z namespaces/openshift-marketplace/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.846233215Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.846237665Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.846279928Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.846353612Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.846366073Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.846397455Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.84649479Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.846579545Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.84666507Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.846743375Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.846781067Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.846946987Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.847123507Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.847264596Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.847467227Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.84767319Z namespaces/openshift-monitoring/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.847813808Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.847875182Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.847912554Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.847978508Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.848002439Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.848161228Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.848605355Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.848984517Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.849214111Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.849249313Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.849298346Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.849327847Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.849420583Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.849485267Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.84954455Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.849616925Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.849653107Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.84972104Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.85294181Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.853088849Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.854214116Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.85429345Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.855338402Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.85547406Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.856082016Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.856250076Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.856284948Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.856436537Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.856470929Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.856523752Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.856561874Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.856675691Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.856780707Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.856822569Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.856854681Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.85699504Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.857028931Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.857123497Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.857158269Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.857289377Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.857325299Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.857434935Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.857609036Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.857700171Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.858032801Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.858173189Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.858305977Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.858398372Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.858510459Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.858599234Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.858669898Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.858736982Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.858837528Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.858921123Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.859007238Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.859114094Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.859196839Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.859350779Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.859436813Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.859530709Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.859633685Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.859734811Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.859835377Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.859903431Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.859934923Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.859993786Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.860001037Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.86004818Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.860312815Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.860538589Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.860706538Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.860975914Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.861188027Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.861220539Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.861326985Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.861336096Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.86141374Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.86157864Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.86158583Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.86159046Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.861638273Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.861763311Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.861835355Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.861864177Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.861873157Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.861877717Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.86191696Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862017976Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.86209219Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862135733Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862143703Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862148224Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862190946Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862284012Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862357816Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862390708Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862397318Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862404279Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862451581Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862546837Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862617871Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862661624Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862668284Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862672815Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862732158Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862823153Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862898618Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.86292963Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862953631Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862958231Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.862968052Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.863086549Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.863145932Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.863186235Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.863194565Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.863208126Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.863245378Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.863334714Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.863413798Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.863453741Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-85sm8/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.863483202Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-85sm8/cluster-monitoring-operator-64786d9cbb-85sm8.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.863595899Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-85sm8/cluster-monitoring-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.863603339Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-85sm8/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.86360917Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-85sm8/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.863650532Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-85sm8/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.864536594Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-85sm8/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.864620109Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-85sm8/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.864649971Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.864730966Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/kube-state-metrics-7c9dbb8b7d-v2h4j.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.864832042Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/kube-rbac-proxy-main/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.864839602Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.864844222Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.864884815Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.86497852Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.865050235Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.865077426Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/kube-rbac-proxy-self/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.865084386Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.865088867Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.86513682Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.865230925Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.865299879Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.865330121Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/kube-state-metrics/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.865336732Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.865346962Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.865391735Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.865571835Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.865675192Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-v2h4j/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.865725615Z namespaces/openshift-monitoring/pods/metrics-server-86bddcdcdf-lprv2/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.865780458Z namespaces/openshift-monitoring/pods/metrics-server-86bddcdcdf-lprv2/metrics-server-86bddcdcdf-lprv2.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.865869093Z namespaces/openshift-monitoring/pods/metrics-server-86bddcdcdf-lprv2/metrics-server/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.865876433Z namespaces/openshift-monitoring/pods/metrics-server-86bddcdcdf-lprv2/metrics-server/metrics-server/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.865880894Z namespaces/openshift-monitoring/pods/metrics-server-86bddcdcdf-lprv2/metrics-server/metrics-server/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.865922706Z namespaces/openshift-monitoring/pods/metrics-server-86bddcdcdf-lprv2/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866017852Z namespaces/openshift-monitoring/pods/metrics-server-86bddcdcdf-lprv2/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866091866Z namespaces/openshift-monitoring/pods/metrics-server-86bddcdcdf-lprv2/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866143609Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-wzr47/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866173481Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-wzr47/monitoring-plugin-d5dcff7cf-wzr47.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866261416Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-wzr47/monitoring-plugin/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866269036Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-wzr47/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866273497Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-wzr47/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866309349Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-wzr47/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866412285Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-wzr47/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866481939Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-wzr47/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866528622Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866570084Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/node-exporter-9vj4l.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866676291Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/init-textfile/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866701902Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/init-textfile/init-textfile/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866706552Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/init-textfile/init-textfile/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866743544Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866821649Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866891903Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866925955Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866935316Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866939786Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.866997619Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.867088835Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.867157959Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.867195021Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/node-exporter/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.867204592Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/node-exporter/node-exporter/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.867209182Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/node-exporter/node-exporter/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.867268685Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.867384432Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.867449306Z namespaces/openshift-monitoring/pods/node-exporter-9vj4l/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.86750878Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.867528421Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/node-exporter-sldgn.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.867657398Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/init-textfile/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.867664599Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/init-textfile/init-textfile/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.867669169Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/init-textfile/init-textfile/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.867735473Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.867814198Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.867883702Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.867912213Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.867917364Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.867921854Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.867968657Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868062112Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868130426Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868170248Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/node-exporter/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868177029Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/node-exporter/node-exporter/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868181529Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/node-exporter/node-exporter/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868219962Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868352259Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868423854Z namespaces/openshift-monitoring/pods/node-exporter-sldgn/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868460746Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868500038Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/node-exporter-zl5tf.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868621545Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/init-textfile/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868628946Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/init-textfile/init-textfile/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868633406Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/init-textfile/init-textfile/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868671098Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868800696Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868863689Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868899152Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868910922Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868919553Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.868958145Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.869051881Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.869120975Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.869175498Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/node-exporter/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.869193309Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/node-exporter/node-exporter/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.869197989Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/node-exporter/node-exporter/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.869220001Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.869325147Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.869393061Z namespaces/openshift-monitoring/pods/node-exporter-zl5tf/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.869421742Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.869478206Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/openshift-state-metrics-54cf96dfcf-45gjw.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.869607463Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/kube-rbac-proxy-main/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.869615174Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.869619764Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.869666407Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.869780423Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.869850518Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.869879599Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/kube-rbac-proxy-self/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.86988478Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.86989284Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.869937873Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.870025608Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.870106783Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.870122894Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/openshift-state-metrics/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.870131484Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.870161916Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.870211649Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.870313945Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.870387239Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-45gjw/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.870445663Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.870512647Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.870704428Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.870727059Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.87073539Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.870793563Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.870898129Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.870966524Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871022147Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871035208Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871041148Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.87107154Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871160555Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871229899Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871257401Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871264091Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871271031Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871310754Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871404939Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871471643Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871524306Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871537467Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871542947Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871550348Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871635903Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.87175243Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871774381Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871780661Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871785022Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871833635Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.87192042Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.871989864Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.872020606Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.872028546Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.872044847Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.872063458Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.87294794Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.873016104Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.873053737Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.873070638Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.873076468Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.87311403Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.873214536Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.8732845Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.873315232Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-mnpck/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.873371215Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-mnpck/prometheus-operator-567f7b8b7c-mnpck.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.873467681Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-mnpck/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.873481872Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-mnpck/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.873489342Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-mnpck/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.873499733Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-mnpck/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.873603659Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-mnpck/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.873669373Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-mnpck/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.873712135Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-mnpck/prometheus-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.873735147Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-mnpck/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.873747128Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-mnpck/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.87378378Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-mnpck/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.874047285Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-mnpck/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.874119259Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-mnpck/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.874150891Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-djgk4/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.874198564Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-djgk4/prometheus-operator-admission-webhook-d559d66d5-djgk4.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.874280999Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-djgk4/prometheus-operator-admission-webhook/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.87428912Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-djgk4/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.87429714Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-djgk4/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.874336052Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-djgk4/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.874435388Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-djgk4/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.874498052Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-djgk4/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.874532204Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.874588097Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/telemeter-client-746978fcf7-tlq6x.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.874758807Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.874769848Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.874778168Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.874825351Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.874918067Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.874990571Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.875027173Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/reload/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.875033513Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/reload/reload/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.875038084Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/reload/reload/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.875083596Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/reload/reload/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.875178502Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.875252006Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/reload/reload/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.875290268Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/telemeter-client/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.875298719Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/telemeter-client/telemeter-client/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.875307789Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.875354182Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.875501241Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.875571305Z namespaces/openshift-monitoring/pods/telemeter-client-746978fcf7-tlq6x/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.875602387Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.875700243Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/thanos-querier-687687fc7d-drjss.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.875834891Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy-metrics/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.875842441Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.875847171Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.875883863Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.87600247Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.876074945Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.87616843Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy-rules/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.876175881Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.876180581Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.876225484Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.876315179Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.876392173Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.876423645Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy-web/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.876432826Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.876445707Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.876490069Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.876591545Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.87667297Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.876732944Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.876747045Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.876752285Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.876789507Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.876882402Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.876955057Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.876992249Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/prom-label-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.876999239Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.87700394Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.877034151Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.877124397Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.877196801Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.877240574Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/thanos-query/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.877249594Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/thanos-query/thanos-query/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.877255055Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/thanos-query/thanos-query/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.877291597Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.877405533Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.877479248Z namespaces/openshift-monitoring/pods/thanos-querier-687687fc7d-drjss/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.87752057Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.877561982Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.877605615Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.877653468Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.877798547Z namespaces/openshift-multus/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.877834779Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.877921444Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.877953516Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.878006959Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.878041271Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.878268374Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.878350759Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.878438904Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.878486917Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.878518319Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.87870114Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.878828097Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.878897791Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.878948734Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.878980646Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.879037749Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.879120114Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.879155047Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.87920903Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.879294625Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.879361619Z namespaces/openshift-multus/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.879414052Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.8795585Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.879649065Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.880357987Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.880489385Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.881186826Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.881283782Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.881432451Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.881490844Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.881544337Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.881691556Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.881782522Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.881885737Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.881986414Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.882020996Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.882123882Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.882209657Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.882307542Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.882359925Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.882392177Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.882466902Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.882545046Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.88261123Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.882652383Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.882734747Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.882785861Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.882867185Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.88295155Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.883015864Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.883059467Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.88311905Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.883172913Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.883225856Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.88327996Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.883326262Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.883335913Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.883403337Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.883560366Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.88378506Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.883942669Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.884171402Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.884365074Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.884375775Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.884424137Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/multus-additional-cni-plugins-h2pdn.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.884579066Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/bond-cni-plugin/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.884589467Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.884594527Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.88464243Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.884790429Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.884869933Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.884914136Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/cni-plugins/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.884923267Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/cni-plugins/cni-plugins/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.884928457Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.88497382Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.885080426Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.88515729Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.885189682Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/egress-router-binary-copy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.885199213Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.885204253Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.885245276Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.885346942Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.885423546Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.885465029Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.885472689Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.885514402Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.885564474Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.885649819Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.885752576Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.885791078Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/routeoverride-cni/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.885796758Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.885801379Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.88582798Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.885936427Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.886038053Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.886077515Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/whereabouts-cni-bincopy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.886085395Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.886090065Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.886132178Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.886226333Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.886302568Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.88633222Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/whereabouts-cni/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.88634116Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.886345881Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.886388733Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.886485549Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.886561643Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-h2pdn/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.886600366Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.886642068Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/multus-additional-cni-plugins-jn5nv.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.886814428Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/bond-cni-plugin/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.886825929Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.886830649Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.886864751Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.886960807Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887033811Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887069403Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/cni-plugins/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887076784Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/cni-plugins/cni-plugins/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887081504Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887113006Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887205741Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887280746Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887316508Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/egress-router-binary-copy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887323938Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887328698Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887378612Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887466767Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887538601Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887576413Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887584424Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887590344Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887628556Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887809897Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887889302Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887919293Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/routeoverride-cni/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887926464Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887931224Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.887972987Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888064772Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888141486Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888171978Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/whereabouts-cni-bincopy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888180989Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888187109Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888224161Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888314937Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888388491Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888429214Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/whereabouts-cni/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888437874Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888443534Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888489137Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888588853Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888663317Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-jn5nv/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888727201Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888777414Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/multus-additional-cni-plugins-sppzx.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888914672Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/bond-cni-plugin/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888923793Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888928473Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.888959635Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.889060151Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.889131175Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.889204169Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/cni-plugins/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.88921562Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/cni-plugins/cni-plugins/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.88922034Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.889262753Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.889348298Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.889421912Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.889454154Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/egress-router-binary-copy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.889461504Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.889466205Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.889510357Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.889606853Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.889691468Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.88972576Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.88973197Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.889736891Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.889785374Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.889857128Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.889931382Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.889965024Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/routeoverride-cni/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.889972074Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.889977195Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890015847Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890110713Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890178937Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890209279Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/whereabouts-cni-bincopy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890214879Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890219659Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890268212Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890359517Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890436302Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890465864Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/whereabouts-cni/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890472784Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890477264Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890514477Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890604192Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890690717Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sppzx/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890733719Z namespaces/openshift-multus/pods/multus-clchg/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890780472Z namespaces/openshift-multus/pods/multus-clchg/multus-clchg.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890877248Z namespaces/openshift-multus/pods/multus-clchg/kube-multus/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890884808Z namespaces/openshift-multus/pods/multus-clchg/kube-multus/kube-multus/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.890889399Z namespaces/openshift-multus/pods/multus-clchg/kube-multus/kube-multus/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.89092241Z namespaces/openshift-multus/pods/multus-clchg/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.892027836Z namespaces/openshift-multus/pods/multus-clchg/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.89210057Z namespaces/openshift-multus/pods/multus-clchg/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.892130222Z namespaces/openshift-multus/pods/multus-lgsmp/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.892181935Z namespaces/openshift-multus/pods/multus-lgsmp/multus-lgsmp.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.89227773Z namespaces/openshift-multus/pods/multus-lgsmp/kube-multus/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.892292271Z namespaces/openshift-multus/pods/multus-lgsmp/kube-multus/kube-multus/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.892297562Z namespaces/openshift-multus/pods/multus-lgsmp/kube-multus/kube-multus/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.892318013Z namespaces/openshift-multus/pods/multus-lgsmp/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.893134331Z namespaces/openshift-multus/pods/multus-lgsmp/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.893203225Z namespaces/openshift-multus/pods/multus-lgsmp/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.893224796Z namespaces/openshift-multus/pods/multus-n5z72/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.89327896Z namespaces/openshift-multus/pods/multus-n5z72/multus-n5z72.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.893370515Z namespaces/openshift-multus/pods/multus-n5z72/kube-multus/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.893382586Z namespaces/openshift-multus/pods/multus-n5z72/kube-multus/kube-multus/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.893387866Z namespaces/openshift-multus/pods/multus-n5z72/kube-multus/kube-multus/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.893413217Z namespaces/openshift-multus/pods/multus-n5z72/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.89379372Z namespaces/openshift-multus/pods/multus-n5z72/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.893872195Z namespaces/openshift-multus/pods/multus-n5z72/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.893914737Z namespaces/openshift-multus/pods/network-metrics-daemon-6vs4n/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.893947259Z namespaces/openshift-multus/pods/network-metrics-daemon-6vs4n/network-metrics-daemon-6vs4n.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.894039034Z namespaces/openshift-multus/pods/network-metrics-daemon-6vs4n/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.894047975Z namespaces/openshift-multus/pods/network-metrics-daemon-6vs4n/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.894053075Z namespaces/openshift-multus/pods/network-metrics-daemon-6vs4n/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.894096588Z namespaces/openshift-multus/pods/network-metrics-daemon-6vs4n/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.894194554Z namespaces/openshift-multus/pods/network-metrics-daemon-6vs4n/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.894263538Z namespaces/openshift-multus/pods/network-metrics-daemon-6vs4n/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.89430076Z namespaces/openshift-multus/pods/network-metrics-daemon-6vs4n/network-metrics-daemon/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.89430965Z namespaces/openshift-multus/pods/network-metrics-daemon-6vs4n/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.894314661Z namespaces/openshift-multus/pods/network-metrics-daemon-6vs4n/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.894355713Z namespaces/openshift-multus/pods/network-metrics-daemon-6vs4n/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.894521223Z namespaces/openshift-multus/pods/network-metrics-daemon-6vs4n/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.894595767Z namespaces/openshift-multus/pods/network-metrics-daemon-6vs4n/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.89463502Z namespaces/openshift-multus/pods/network-metrics-daemon-j8bqp/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.894667061Z namespaces/openshift-multus/pods/network-metrics-daemon-j8bqp/network-metrics-daemon-j8bqp.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.894768277Z namespaces/openshift-multus/pods/network-metrics-daemon-j8bqp/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.894780848Z namespaces/openshift-multus/pods/network-metrics-daemon-j8bqp/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.894785538Z namespaces/openshift-multus/pods/network-metrics-daemon-j8bqp/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.8948211Z namespaces/openshift-multus/pods/network-metrics-daemon-j8bqp/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.894909286Z namespaces/openshift-multus/pods/network-metrics-daemon-j8bqp/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.89498061Z namespaces/openshift-multus/pods/network-metrics-daemon-j8bqp/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.895019982Z namespaces/openshift-multus/pods/network-metrics-daemon-j8bqp/network-metrics-daemon/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.895028523Z namespaces/openshift-multus/pods/network-metrics-daemon-j8bqp/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.895035643Z namespaces/openshift-multus/pods/network-metrics-daemon-j8bqp/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.895082736Z namespaces/openshift-multus/pods/network-metrics-daemon-j8bqp/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.895283428Z namespaces/openshift-multus/pods/network-metrics-daemon-j8bqp/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.895354922Z namespaces/openshift-multus/pods/network-metrics-daemon-j8bqp/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.895384854Z namespaces/openshift-multus/pods/network-metrics-daemon-mjfls/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.895419046Z namespaces/openshift-multus/pods/network-metrics-daemon-mjfls/network-metrics-daemon-mjfls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.895501831Z namespaces/openshift-multus/pods/network-metrics-daemon-mjfls/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.895510291Z namespaces/openshift-multus/pods/network-metrics-daemon-mjfls/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.895514771Z namespaces/openshift-multus/pods/network-metrics-daemon-mjfls/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.895553243Z namespaces/openshift-multus/pods/network-metrics-daemon-mjfls/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.895646639Z namespaces/openshift-multus/pods/network-metrics-daemon-mjfls/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.895807678Z namespaces/openshift-multus/pods/network-metrics-daemon-mjfls/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.895847151Z namespaces/openshift-multus/pods/network-metrics-daemon-mjfls/network-metrics-daemon/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.895854631Z namespaces/openshift-multus/pods/network-metrics-daemon-mjfls/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.895864302Z namespaces/openshift-multus/pods/network-metrics-daemon-mjfls/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.895909675Z namespaces/openshift-multus/pods/network-metrics-daemon-mjfls/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.896028062Z namespaces/openshift-multus/pods/network-metrics-daemon-mjfls/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.896097466Z namespaces/openshift-multus/pods/network-metrics-daemon-mjfls/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.896127877Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.89617323Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.896207562Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.896214322Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.896266166Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.896355821Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.896403744Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.896443576Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.896530111Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.896583864Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.896625097Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.896689841Z namespaces/openshift-must-gather-8hm2m/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.896701091Z namespaces/openshift-must-gather-8hm2m/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.896705142Z namespaces/openshift-must-gather-8hm2m/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.896753074Z namespaces/openshift-must-gather-8hm2m/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.896908254Z namespaces/openshift-must-gather-8hm2m/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.897083344Z namespaces/openshift-must-gather-8hm2m/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.897218132Z namespaces/openshift-must-gather-8hm2m/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.897419874Z namespaces/openshift-must-gather-8hm2m/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.897611155Z namespaces/openshift-network-console/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.897648237Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.897737523Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.897767584Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.897823397Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.897857339Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.897947025Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.898059962Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.898162308Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.898190799Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.898243602Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.898315417Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.898355189Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.898404272Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.898437764Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.898522139Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.89855468Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.898606124Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.898700139Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.898758332Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.898801095Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.898917862Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.899001507Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.899118754Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.899198258Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.899337697Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.899464214Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.899599422Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.899655065Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.899718569Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.899759342Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.899812245Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.899866558Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.89990629Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.899959703Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.899988025Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.900042528Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.90007083Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.900158595Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.90023743Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.900284683Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.900327315Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.900368928Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.90041537Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.900460693Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.900467613Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.900507726Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.900694227Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.900904889Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.901042587Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.901238509Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.9014244Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.901435081Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-k76n2/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.901452451Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-k76n2/networking-console-plugin-6bffdf9778-k76n2.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.901546447Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-k76n2/networking-console-plugin/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.901555077Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-k76n2/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.901559788Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-k76n2/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.901587299Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-k76n2/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.901699716Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-k76n2/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.901773461Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-k76n2/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.901805292Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.901836004Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.901896608Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.90192821Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.901975122Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.902025445Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.902078939Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.90210805Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.902162593Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.902192195Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.902307162Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.902401698Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.902499653Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.902539146Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.902587209Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.902648572Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.902695585Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.90277785Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.902811122Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.902848614Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.902892747Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.902978241Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.903038595Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.903072377Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.903170663Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.903199665Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.903249428Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.903349823Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.903437899Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.903822081Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.903926637Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.90414289Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.904229195Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.904337052Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.904404746Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.904442628Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.904518932Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.904557845Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.904667811Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.904745956Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.904783758Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.904848592Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.904895305Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.904947528Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.90498503Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.905067985Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.905144599Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.905191292Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.905217664Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.905281787Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.905308529Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.905368882Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.905401584Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.905452847Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.905461098Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.90549415Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.905644069Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.905851171Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.905986979Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.906188991Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.906381482Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.906389233Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-ktssk/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.906426235Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-ktssk/network-check-source-989f8b8dc-ktssk.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.906502929Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-ktssk/check-endpoints/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.90651049Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-ktssk/check-endpoints/check-endpoints/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.90651453Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-ktssk/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.906553262Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-ktssk/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.906647398Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-ktssk/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.906753374Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-ktssk/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.906789346Z namespaces/openshift-network-diagnostics/pods/network-check-target-gvkfp/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.906820528Z namespaces/openshift-network-diagnostics/pods/network-check-target-gvkfp/network-check-target-gvkfp.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.906903043Z namespaces/openshift-network-diagnostics/pods/network-check-target-gvkfp/network-check-target-container/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.906910533Z namespaces/openshift-network-diagnostics/pods/network-check-target-gvkfp/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.906914484Z namespaces/openshift-network-diagnostics/pods/network-check-target-gvkfp/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.906945726Z namespaces/openshift-network-diagnostics/pods/network-check-target-gvkfp/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907035251Z namespaces/openshift-network-diagnostics/pods/network-check-target-gvkfp/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907100415Z namespaces/openshift-network-diagnostics/pods/network-check-target-gvkfp/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907133026Z namespaces/openshift-network-diagnostics/pods/network-check-target-kkslg/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907180009Z namespaces/openshift-network-diagnostics/pods/network-check-target-kkslg/network-check-target-kkslg.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907253364Z namespaces/openshift-network-diagnostics/pods/network-check-target-kkslg/network-check-target-container/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907261424Z namespaces/openshift-network-diagnostics/pods/network-check-target-kkslg/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907265494Z namespaces/openshift-network-diagnostics/pods/network-check-target-kkslg/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907299846Z namespaces/openshift-network-diagnostics/pods/network-check-target-kkslg/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907388072Z namespaces/openshift-network-diagnostics/pods/network-check-target-kkslg/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907454365Z namespaces/openshift-network-diagnostics/pods/network-check-target-kkslg/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907483877Z namespaces/openshift-network-diagnostics/pods/network-check-target-p5hbx/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.90752925Z namespaces/openshift-network-diagnostics/pods/network-check-target-p5hbx/network-check-target-p5hbx.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907600424Z namespaces/openshift-network-diagnostics/pods/network-check-target-p5hbx/network-check-target-container/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907609565Z namespaces/openshift-network-diagnostics/pods/network-check-target-p5hbx/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907614425Z namespaces/openshift-network-diagnostics/pods/network-check-target-p5hbx/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907639056Z namespaces/openshift-network-diagnostics/pods/network-check-target-p5hbx/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907755943Z namespaces/openshift-network-diagnostics/pods/network-check-target-p5hbx/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907827338Z namespaces/openshift-network-diagnostics/pods/network-check-target-p5hbx/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907860269Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907899812Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907952475Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907960475Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.907995508Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.908081882Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.908127305Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.908169438Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.908250283Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.908300135Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.908345368Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.908385471Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.908442084Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.908497207Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.90854349Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.908600393Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.908628855Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.908747792Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.908829697Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.908909171Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.908942373Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.908994676Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.909042409Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.909084672Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.909164926Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.909210389Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.909237871Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.909327356Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.909360778Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.909367168Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.909413661Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.909465114Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.909508607Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.909602692Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.909698428Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.909801574Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.909879728Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.909957773Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.910098972Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.910204528Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.910253821Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.910301884Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.910348916Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.910380928Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.910426601Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.910462223Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.910499415Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.910551298Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.910632123Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.910741659Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.910789282Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.910825214Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.910875887Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.910912049Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.910962312Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.910969423Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.911010655Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.911163704Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.911339635Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.911480533Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.911667274Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.911875026Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.911909458Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.911961501Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.911968602Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.912016425Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.912065908Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.912093209Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.912149822Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.912198865Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.912242698Z namespaces/openshift-network-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.912290731Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.912353165Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.912386776Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.912423379Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.912472792Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.912587768Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.912672423Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.91277995Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.912825952Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.912867845Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.912934459Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.91296465Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.913017684Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.913044945Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.91313358Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.913187594Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.913222786Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.913302761Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.913342143Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.913348393Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.913391616Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.913444599Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.913488791Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.913608608Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.913702564Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.913957869Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.914033504Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.914207264Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.914294029Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.914420416Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.91447235Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.914512302Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.914599727Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.914660671Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.914729515Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.914778237Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.914830481Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.914877533Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.914903745Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.914961878Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.915002021Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.915080465Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.9151577Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.915204413Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.915242555Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.915298638Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.915342211Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.915384703Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.915433806Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.915485089Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.915513281Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.915569504Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.915576495Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.915617347Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.915795257Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.915978958Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.916114896Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.916309928Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.916491789Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.916498739Z namespaces/openshift-network-operator/pods/iptables-alerter-lqxtl/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.91652578Z namespaces/openshift-network-operator/pods/iptables-alerter-lqxtl/iptables-alerter-lqxtl.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.916604895Z namespaces/openshift-network-operator/pods/iptables-alerter-lqxtl/iptables-alerter/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.916614596Z namespaces/openshift-network-operator/pods/iptables-alerter-lqxtl/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.916621726Z namespaces/openshift-network-operator/pods/iptables-alerter-lqxtl/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.916674229Z namespaces/openshift-network-operator/pods/iptables-alerter-lqxtl/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.916801097Z namespaces/openshift-network-operator/pods/iptables-alerter-lqxtl/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.916875221Z namespaces/openshift-network-operator/pods/iptables-alerter-lqxtl/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.916906603Z namespaces/openshift-network-operator/pods/iptables-alerter-w8gzj/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.916949856Z namespaces/openshift-network-operator/pods/iptables-alerter-w8gzj/iptables-alerter-w8gzj.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.91702115Z namespaces/openshift-network-operator/pods/iptables-alerter-w8gzj/iptables-alerter/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.91702847Z namespaces/openshift-network-operator/pods/iptables-alerter-w8gzj/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.91703243Z namespaces/openshift-network-operator/pods/iptables-alerter-w8gzj/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.917065332Z namespaces/openshift-network-operator/pods/iptables-alerter-w8gzj/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.917150067Z namespaces/openshift-network-operator/pods/iptables-alerter-w8gzj/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.917215981Z namespaces/openshift-network-operator/pods/iptables-alerter-w8gzj/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.917256674Z namespaces/openshift-network-operator/pods/iptables-alerter-z62br/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.917287016Z namespaces/openshift-network-operator/pods/iptables-alerter-z62br/iptables-alerter-z62br.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.91735968Z namespaces/openshift-network-operator/pods/iptables-alerter-z62br/iptables-alerter/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.91736711Z namespaces/openshift-network-operator/pods/iptables-alerter-z62br/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.917371141Z namespaces/openshift-network-operator/pods/iptables-alerter-z62br/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.917410093Z namespaces/openshift-network-operator/pods/iptables-alerter-z62br/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.917491308Z namespaces/openshift-network-operator/pods/iptables-alerter-z62br/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.917559812Z namespaces/openshift-network-operator/pods/iptables-alerter-z62br/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.917588603Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.917628566Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.917672688Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.917744183Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.917790045Z namespaces/openshift-node/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.917797806Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.917801656Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.917838118Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.917984887Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.918168828Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.918303496Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.918499877Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.918771013Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.918816506Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.918876219Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.918907991Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.918962174Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.918993186Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.919084122Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.919161976Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.919260662Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.919305264Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.919333016Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.9194047Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.919448333Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.919536048Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.91957188Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.919619623Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.919695738Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.919801874Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.919836336Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.919937062Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.920018766Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.920110802Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.920189507Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.920269551Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.920389068Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.92058769Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.920653804Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.920704707Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.920779831Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.920809993Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.920863287Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.920910099Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.920987924Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.921069588Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.921114171Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.921153084Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.921200876Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.921228518Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.921297992Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.921339384Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.921418679Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.9214278Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.921470292Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.921642652Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.921850234Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.921978812Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.922174544Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.922362435Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.922390666Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.92245657Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.922499723Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.922562007Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.922581418Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.922640631Z namespaces/openshift-operators/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.922650832Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.922655652Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.922663262Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.922846623Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.923023624Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.923159522Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.923346133Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.923526144Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.923553155Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.923613589Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.923654321Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.923741956Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.923784319Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.923827801Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.923876394Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.924051075Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.924127839Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.924208084Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.924239335Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.924283608Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.924417336Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.924442118Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.924497111Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.924531473Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.924627718Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.924663331Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.924732175Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.9248155Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.924861362Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.924871333Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.924910225Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.924961008Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.925006211Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.925199352Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.925283657Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.925642348Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.925786167Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.926264415Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.92634898Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.92651484Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.926571223Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.926590474Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.926705191Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.926834379Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.926878011Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.926970307Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.927015139Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.927052801Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.927111105Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.927140407Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.927216561Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.927248653Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.927302636Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.927309777Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.927347999Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.927401842Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.927431214Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.927526279Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.927602504Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.927641396Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.927690199Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.927838118Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.9278664Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.927953334Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.927988087Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.92804177Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.92804923Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.928086793Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.928173698Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.928206929Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.928257483Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.928301885Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.928308586Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.928349978Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.928587372Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.928788394Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.928925602Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.929123363Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.929312315Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.929319635Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.929380909Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/ovnkube-node-82n9w.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.92956328Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/kube-rbac-proxy-node/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.92957216Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.92957719Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.929611262Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.929756561Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.929828155Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.929845216Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.929853306Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.929857817Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.92990575Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930045728Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930113802Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930151804Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/nbdb/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930165755Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/nbdb/nbdb/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930174236Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/nbdb/nbdb/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930193637Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/nbdb/nbdb/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930301003Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930358907Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930378978Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/northd/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930387568Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/northd/northd/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930396509Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/northd/northd/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930436901Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/northd/northd/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930542097Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930610791Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/northd/northd/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930640873Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/ovn-acl-logging/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930647493Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930651954Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930696626Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930808713Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930855596Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/ovn-controller/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930862376Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/ovn-controller/ovn-controller/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930866916Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.930909789Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.931126942Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.931197466Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.931237958Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/ovnkube-controller/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.931247129Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.931251719Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.931281971Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.932705985Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.932780229Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.932888005Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/sbdb/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.932896616Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/sbdb/sbdb/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.932903517Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/sbdb/sbdb/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.932935728Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/sbdb/sbdb/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.933033274Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.933099868Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-82n9w/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.93313977Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.933205764Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ovnkube-node-b2hcz.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.933352693Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/kube-rbac-proxy-node/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.933360644Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.933365234Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.933395875Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.933525443Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.933595107Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.933624919Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.933631809Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.93363624Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.933673872Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.933830991Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.933907136Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.933941128Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/nbdb/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.933948158Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/nbdb/nbdb/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.933952598Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/nbdb/nbdb/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.933995071Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/nbdb/nbdb/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.934084876Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.93415367Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.934181192Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/northd/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.934188402Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/northd/northd/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.934192922Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/northd/northd/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.934222524Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/northd/northd/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.934332151Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.934400975Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/northd/northd/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.934435407Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ovn-acl-logging/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.934443107Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.934447668Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.93448322Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.934574345Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.934643479Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.934705003Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ovn-controller/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.934717204Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ovn-controller/ovn-controller/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.934721854Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.934748035Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.935120407Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.935192392Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.935230154Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ovnkube-controller/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.935237304Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.935241744Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.935278257Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.937413932Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.937491987Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.937510098Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/sbdb/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.937517239Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/sbdb/sbdb/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.937522149Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/sbdb/sbdb/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.937558601Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/sbdb/sbdb/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.937660837Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.937754352Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-b2hcz/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.937772674Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.937859009Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ovnkube-node-v7kmn.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938009808Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/kube-rbac-proxy-node/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938024439Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938029629Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.9380497Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938182788Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938255132Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938290714Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938298725Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938303835Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938337607Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938457344Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938524888Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938569331Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/nbdb/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938577851Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/nbdb/nbdb/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938582351Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/nbdb/nbdb/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938604613Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/nbdb/nbdb/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.93872825Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938803224Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938833906Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/northd/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938841507Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/northd/northd/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938846097Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/northd/northd/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.938886299Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/northd/northd/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.93907197Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.939135784Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/northd/northd/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.939166206Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ovn-acl-logging/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.939173266Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.939181107Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.939218489Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.939313915Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.939395429Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.93941307Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ovn-controller/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.939422071Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ovn-controller/ovn-controller/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.939428731Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.939465733Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.939778182Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.939851536Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.939885508Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ovnkube-controller/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.939892459Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.939896999Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.939932001Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.941650912Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.941737647Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.941759439Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/sbdb/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.941767239Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/sbdb/sbdb/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.941771969Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/sbdb/sbdb/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.941822783Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/sbdb/sbdb/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.941918938Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.941992193Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v7kmn/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.942012404Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.942074497Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.942129831Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.942139271Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.942151852Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.942262498Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.942386296Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.942420898Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.9424593Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.942558226Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.942641101Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.942689434Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.942758728Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.942792Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.94279904Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.942810031Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.942860184Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.942913267Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.942922068Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.942947109Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.943157531Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.943378504Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.943552754Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.943839791Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.944052884Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.944084746Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.9441606Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.944191912Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.944247755Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.944291978Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.944373063Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.944483089Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.944590646Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.944622238Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.944675711Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.944734774Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.944778477Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.944864152Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.944911745Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.944965568Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.945033632Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.945077264Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.945089145Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.945125287Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.945179301Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.945208172Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.945331379Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.945415664Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.945612126Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.945694411Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.94584397Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.945963487Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.946148488Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.946196961Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.946257254Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.946290856Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.946324368Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.946382881Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.946411153Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.946511949Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.946579323Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.946625786Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.946670469Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.946743363Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.946784805Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.946838578Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.946882761Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.946926593Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.946935014Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.946960006Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.947117945Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.94738038Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.947520839Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.947741012Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.947938303Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.947945804Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-8f2m8/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.947975125Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-8f2m8/service-ca-operator-5d46d9ffd4-8f2m8.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.94805862Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-8f2m8/service-ca-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.948066271Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-8f2m8/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.948070911Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-8f2m8/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.948107123Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-8f2m8/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.948270733Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-8f2m8/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.948448063Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.948469955Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.948525948Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.948578871Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.948621344Z namespaces/openshift-service-ca/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.948662516Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.948751291Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.948778463Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.948838126Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.948867768Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.948965104Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.949062719Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.949163416Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.949201808Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.94923515Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.949292903Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.949330335Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.94941698Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.949453653Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.949500505Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.9495826Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.949632753Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.949640664Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.949664995Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.94975459Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.949776022Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.949889358Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.949972163Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.95008493Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.950164324Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.950277431Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.950402649Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.950564688Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.9506021Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.950656153Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.950727508Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.95076165Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.950816433Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.950852215Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.95093932Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.951022695Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.951062677Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.95110064Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.951139822Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.951191315Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.951245578Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.951252969Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.951296051Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.95144605Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.951628731Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.951888626Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.952106759Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.95229796Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.952305501Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-5nvlr/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.952347553Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-5nvlr/service-ca-758f599c5-5nvlr.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.952435969Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-5nvlr/service-ca-controller/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.952450569Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-5nvlr/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.95245708Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-5nvlr/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.952476701Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-5nvlr/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.952750037Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-5nvlr/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.952825981Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-5nvlr/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.952870914Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.952902706Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.952959689Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.952995351Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.953060855Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.953104608Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.953176602Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.953201774Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.953253287Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.9533088Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.953399975Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.95347477Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.953557485Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.953614378Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.953669851Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.953738095Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.953785608Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.953877494Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.953910296Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.953967939Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.954050364Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.954083846Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.954140819Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.954242885Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.9543315Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.954416515Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.9544992Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.954583515Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.954738014Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.954906284Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.954984659Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.955038972Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.955081935Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.955132408Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.9551831Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.955239474Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.955312938Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.955400903Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.955445816Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.955492819Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.955532341Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.955582474Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.955641898Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.955654588Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.955666919Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.955866751Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.956054322Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.95619738Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.956408433Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.956624776Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.956654927Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.956748533Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.956792755Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.956843529Z namespaces/openshift/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.956885571Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.956951195Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.956988417Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.95704156Z namespaces/openshift/apps/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.957078522Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.957171448Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.957348938Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.957439104Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.957487496Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.957518108Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.957578772Z namespaces/openshift/batch/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.957637195Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.957750622Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.957792425Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.957840397Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.957929933Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.957979685Z namespaces/openshift/core/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.958031999Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.958136415Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.95821839Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.958305155Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.958392Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.958477225Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.958607833Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.958809585Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.958864998Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.958925551Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.958944672Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.959002356Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.960308193Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.960340565Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.960446481Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.960535236Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.960668064Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.960810192Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.960903968Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.961061697Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.961237718Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.961371776Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.961473382Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.961577118Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.961729657Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.961839163Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.961968471Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.962070037Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.962160462Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.962269378Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.962390536Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.962511093Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.962616429Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.962748117Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.962860533Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.962960899Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.963065235Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.963171482Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.963277498Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.963381714Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.963497611Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.963610157Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.963739515Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.963860782Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.963976619Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.964068875Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.964187702Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.964282607Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.964411515Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.964574245Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.96467316Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.96482844Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.964932235Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.965051722Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.965206862Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.965302407Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.965400233Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.965530921Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.965663839Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.965807037Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.965926824Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.96602591Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.966119365Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.966216661Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.966340098Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.966465046Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.966587703Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.966733752Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.966858819Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.966962265Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.967058811Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.967193489Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.967270904Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.967298305Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.9673917Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.967475776Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.967641185Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.967671737Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.967782564Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.967833577Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.967879109Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.96788709Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.967937123Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.968087542Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.968260982Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.96839949Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.968618343Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.968842796Z namespaces/openshift/policy/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.968863707Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.968940122Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.969006856Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.969087451Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.969095701Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.969140694Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.969261271Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.969386168Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.969500775Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.969611931Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.969845005Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.969989184Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.970134302Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.970250919Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.970381477Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.970501114Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.970620371Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.97077857Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.970928639Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.971037766Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.971161933Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.971273889Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.971392636Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.971516954Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.9716269Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.971761148Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.971867484Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.971974001Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.972101538Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.972198764Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.97230971Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.972415197Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.972521833Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.972657641Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.972819241Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.972935878Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.973065065Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.973192403Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.973303739Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.973431087Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.973550014Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.97365169Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.973782737Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.973904325Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.974014571Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.974125238Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.974259146Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.974381433Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.974514731Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.974648099Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.974805198Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.974964717Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.975099225Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.975232633Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.97535403Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.975505909Z network_logs/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.975540721Z network_logs/cluster_scale [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.975634667Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.975793446Z network_logs/multi-networkpolicy [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.975879121Z network_logs/net-attach-def [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.975965376Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.976125566Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.976221981Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.979347246Z nodes/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.979374587Z nodes/debug [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.979489424Z nodes/ip-10-0-131-45.ec2.internal/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.979536667Z nodes/ip-10-0-131-45.ec2.internal/cpu_affinities.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.979893898Z nodes/ip-10-0-131-45.ec2.internal/dmesg [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.980183265Z nodes/ip-10-0-131-45.ec2.internal/ethtool_channels [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.98026724Z nodes/ip-10-0-131-45.ec2.internal/ethtool_features [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.980360065Z nodes/ip-10-0-131-45.ec2.internal/ip-10-0-131-45.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.980869435Z nodes/ip-10-0-131-45.ec2.internal/irq_affinities.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.981016274Z nodes/ip-10-0-131-45.ec2.internal/lscpu [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.981149812Z nodes/ip-10-0-131-45.ec2.internal/lspci [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.981254728Z nodes/ip-10-0-131-45.ec2.internal/podresources.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.981364095Z nodes/ip-10-0-131-45.ec2.internal/pods_info.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.981495192Z nodes/ip-10-0-131-45.ec2.internal/proc_cmdline [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.981580337Z nodes/ip-10-0-131-45.ec2.internal/sysinfo.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.983371863Z nodes/ip-10-0-131-45.ec2.internal/sysinfo.tgz [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.983549493Z nodes/ip-10-0-131-57.ec2.internal/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.983594746Z nodes/ip-10-0-131-57.ec2.internal/cpu_affinities.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.984200272Z nodes/ip-10-0-131-57.ec2.internal/dmesg [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.984907464Z nodes/ip-10-0-131-57.ec2.internal/ethtool_channels [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.98501416Z nodes/ip-10-0-131-57.ec2.internal/ethtool_features [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.985137267Z nodes/ip-10-0-131-57.ec2.internal/ip-10-0-131-57.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.986053671Z nodes/ip-10-0-131-57.ec2.internal/irq_affinities.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.986234112Z nodes/ip-10-0-131-57.ec2.internal/lscpu [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.986388051Z nodes/ip-10-0-131-57.ec2.internal/lspci [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.986500367Z nodes/ip-10-0-131-57.ec2.internal/podresources.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.986629565Z nodes/ip-10-0-131-57.ec2.internal/pods_info.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.986788064Z nodes/ip-10-0-131-57.ec2.internal/proc_cmdline [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.986893271Z nodes/ip-10-0-131-57.ec2.internal/sysinfo.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.988763721Z nodes/ip-10-0-131-57.ec2.internal/sysinfo.tgz [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.988955983Z nodes/ip-10-0-142-77.ec2.internal/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.989020186Z nodes/ip-10-0-142-77.ec2.internal/cpu_affinities.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.989307013Z nodes/ip-10-0-142-77.ec2.internal/dmesg [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.98959504Z nodes/ip-10-0-142-77.ec2.internal/ethtool_channels [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.989698206Z nodes/ip-10-0-142-77.ec2.internal/ethtool_features [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.989816603Z nodes/ip-10-0-142-77.ec2.internal/ip-10-0-142-77.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.990127152Z nodes/ip-10-0-142-77.ec2.internal/irq_affinities.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.990294421Z nodes/ip-10-0-142-77.ec2.internal/lscpu [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.990454721Z nodes/ip-10-0-142-77.ec2.internal/lspci [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.990574328Z nodes/ip-10-0-142-77.ec2.internal/podresources.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.990699235Z nodes/ip-10-0-142-77.ec2.internal/pods_info.json [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.990846884Z nodes/ip-10-0-142-77.ec2.internal/proc_cmdline [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.99094799Z nodes/ip-10-0-142-77.ec2.internal/sysinfo.log [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.992941978Z nodes/ip-10-0-142-77.ec2.internal/sysinfo.tgz [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.993132019Z pod_network_connectivity_check/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.993193972Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.993307509Z static-pods/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:38.99331628Z static-pods/kube-apiserver/ [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:39.000830003Z [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:39.000848544Z sent 71,407 bytes received 9,448,395 bytes 19,039,604.00 bytes/sec [must-gather] [must-gather-8t4g8] OUT 2026-07-06T22:24:39.000856784Z total size is 108,498,219 speedup is 11.40 [must-gather] [must-gather ] OUT 2026-07-06T22:24:39.336443247Z namespace/openshift-must-gather-8hm2m deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 45957e43-6e07-4af2-b514-0fb39f986eca [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.23" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-cz7hd [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-cz7hd-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-cz7hd [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 17114058 Jul 6 22:26 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-cz7hd/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts c4dabcb] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-cz7hd/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] f75413a..c4dabcb ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-07-06T22:26:28.881Z","type":3}]