task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-gq7dr-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-gq7dr-kubeconfig [get-kubeconfig] Found admin password secret: cluster-gq7dr-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-gq7dr-password [get-kubeconfig] API Server URL: https://a16937a1f5987447cb9479e5e4ec6ffa-d8d381b5e676cd94.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.bbfed7ef-61c3-4711-ab5f-0e769310c2f8.prod.konfluxeaas.com [clone-repo] auto-sync/master-to-release-v0.17 [clone-repo] https://github.com/opendatahub-io/kserve [clone-repo] Cloning into '/workspace/source'... [clone-repo] Updating files: 79% (2594/3261) Updating files: 80% (2609/3261) Updating files: 81% (2642/3261) Updating files: 82% (2675/3261) Updating files: 83% (2707/3261) Updating files: 84% (2740/3261) Updating files: 85% (2772/3261) Updating files: 86% (2805/3261) Updating files: 87% (2838/3261) Updating files: 88% (2870/3261) Updating files: 89% (2903/3261) Updating files: 90% (2935/3261) Updating files: 91% (2968/3261) Updating files: 92% (3001/3261) Updating files: 93% (3033/3261) Updating files: 94% (3066/3261) Updating files: 95% (3098/3261) Updating files: 96% (3131/3261) Updating files: 97% (3164/3261) Updating files: 98% (3196/3261) Updating files: 99% (3229/3261) Updating files: 100% (3261/3261) Updating files: 100% (3261/3261), done. [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:bf42462edec7da298c1b007872394204ebe10a2c0c0d143725247988fb0a7324 [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:bf42462edec7da298c1b007872394204ebe10a2c0c0d143725247988fb0a7324 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:95b3fcfd6a46997bfc9b2a3a609d19adade819e90aecabd6142db665882198ae [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:95b3fcfd6a46997bfc9b2a3a609d19adade819e90aecabd6142db665882198ae [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:3f8b4d8bbc07de8aec3f29b089d69c216cddd2ca68e10f109f830f4b0fdd9ee1 [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:3f8b4d8bbc07de8aec3f29b089d69c216cddd2ca68e10f109f830f4b0fdd9ee1 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:247952ac232dfdb9198891fa9af76488d402df0a9e8dd7866db1e42afc206909 [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:247952ac232dfdb9198891fa9af76488d402df0a9e8dd7866db1e42afc206909 [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu and not autoscaling and not tracing' 2 llm-d [e2e-llm-inference-service] INFERENCE_POOL_GROUP=inference.networking.k8s.io (detected from OCP 4.21.26) [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1814 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1814 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1814 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 274 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading cryptography (4.5MiB) [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Prepared 101 packages in 1.83s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 101 packages in 475ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.14.1 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.4.2 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==49.0.0 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.20.0rc0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.11.9 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn1==0.6.3 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.13.0 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.32 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==1.2.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.7.0 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 46ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Now using project "kserve" on server "https://a16937a1f5987447cb9479e5e4ec6ffa-d8d381b5e676cd94.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-2 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-77f648d694-5mh9h 1/1 Running 0 41s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-77f648d694-5mh9h condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-55557df8c-w4jlw 1/1 Running 0 46s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-55557df8c-w4jlw condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-56f9798bb4-ttzr7 1/1 Running 0 52s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-56f9798bb4-ttzr7 condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:95b3fcfd6a46997bfc9b2a3a609d19adade819e90aecabd6142db665882198ae [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:247952ac232dfdb9198891fa9af76488d402df0a9e8dd7866db1e42afc206909 [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:bf42462edec7da298c1b007872394204ebe10a2c0c0d143725247988fb0a7324 [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:3f8b4d8bbc07de8aec3f29b089d69c216cddd2ca68e10f109f830f4b0fdd9ee1 [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] === Final params.env [e2e-llm-inference-service] kserve-controller=quay.io/opendatahub/kserve-controller@sha256:95b3fcfd6a46997bfc9b2a3a609d19adade819e90aecabd6142db665882198ae [e2e-llm-inference-service] llmisvc-controller=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:247952ac232dfdb9198891fa9af76488d402df0a9e8dd7866db1e42afc206909 [e2e-llm-inference-service] kserve-agent=quay.io/opendatahub/kserve-agent@sha256:bf42462edec7da298c1b007872394204ebe10a2c0c0d143725247988fb0a7324 [e2e-llm-inference-service] kserve-router=quay.io/opendatahub/kserve-router@sha256:3f8b4d8bbc07de8aec3f29b089d69c216cddd2ca68e10f109f830f4b0fdd9ee1 [e2e-llm-inference-service] kserve-storage-initializer=quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d [e2e-llm-inference-service] kserve-llm-d=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-inference-scheduler=quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-routing-sidecar=quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] kserve-llm-d-uds-tokenizer=quay.io/opendatahub/llm-d-kv-cache:v0.8.0 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2=registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:fc68d623d1bfc36c8cb2fe4a71f19c8578cfb420ce8ce07b20a02c1ee0be0cf3 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-nvidia-cuda-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2=registry.redhat.io/rhaiis/vllm-rocm-rhel9@sha256:d9a48add238cc095fa43eeee17c8c4d104de60c4dc623e0bc7f8c4b53b2b2e97 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-1-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-amd-rocm-fast-2-upstream-version=0.11.0+rhai5 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2=registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-1-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-intel-gaudi-fast-2-upstream-version=0.16.0 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2=registry.redhat.io/rhaiis/vllm-spyre-rhel9@sha256:80ae3e435a5be2c1f117f36599103ab05357917dd6e37f0df6613cb3ac2c13ea [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-1-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-ibm-spyre-fast-2-upstream-version=0.10.2 [e2e-llm-inference-service] kserve-llm-d-latency-predictor-prediction=quay.io/opendatahub/odh-latency-predictor-prediction:odh-stable [e2e-llm-inference-service] kserve-llm-d-latency-predictor-training=quay.io/opendatahub/odh-latency-predictor-training:odh-stable [e2e-llm-inference-service] # TODO update when our changes are introduced in the official image [e2e-llm-inference-service] kube-rbac-proxy=quay.io/opendatahub/odh-kube-auth-proxy@sha256:dcb09fbabd8811f0956ef612a0c9ddd5236804b9bd6548a0647d2b531c9d01b3 [e2e-llm-inference-service] kserve-localmodel-controller=quay.io/opendatahub/odh-kserve-localmodel-controller:odh-stable [e2e-llm-inference-service] kserve-localmodelnode-agent=quay.io/opendatahub/odh-kserve-localmodelnode-agent:odh-stable [e2e-llm-inference-service] ovms-versioning-ubi-micro=registry.redhat.io/ubi9/ubi-micro@sha256:38e934147827349f2b8b11ac9c38d7be23cb8e29f128b1c46e5c7ae54a2d23cd [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-7584bddd84-xc2hz 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-7584bddd84-xc2hz condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-models-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-tls-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-tls-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler-latency-predictor serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tokenizer serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.26) [e2e-llm-inference-service] 🎯 Server version (4.21.26) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-m8rzf 1/1 Running 0 12s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-m8rzf condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-5677ccb49b-4r9gw 1/1 Running 0 11s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-5677ccb49b-4r9gw condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] ⏳ Creating gateway memory ConfigMap for parametersRef (2Gi) [e2e-llm-inference-service] configmap/gateway-proxy-config created [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq 0/1 Running 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: InstallReady). Waiting... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.2 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-56fb6f6856-vrqn9 1/1 Running 0 75s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-56fb6f6856-vrqn9 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-56fb6f6856-vrqn9 1/1 Running 0 85s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-56fb6f6856-vrqn9 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Setting up Jaeger (Helm) for LLMISVC tracing e2e... [e2e-llm-inference-service] ⏳ Installing Jaeger All-in-One (Helm) into namespace observability [e2e-llm-inference-service] [INFO] Installing Helm v3.16.3 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Helm v3.16.3 to /workspace/source/bin/helm [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-gq7dr-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-gq7dr-kubeconfig [e2e-llm-inference-service] version.BuildInfo{Version:"v3.16.3", GitCommit:"cfd07493f46efc9debd9cc1b02a0961186df7fdf", GitTreeState:"clean", GoVersion:"go1.22.7"} [e2e-llm-inference-service] [INFO] Adding Jaeger Helm repository... [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-gq7dr-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-gq7dr-kubeconfig [e2e-llm-inference-service] "jaegertracing" has been added to your repositories [e2e-llm-inference-service] [INFO] Installing Jaeger All-in-One 4.7.0... [e2e-llm-inference-service] WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /credentials/cluster-gq7dr-kubeconfig [e2e-llm-inference-service] WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /credentials/cluster-gq7dr-kubeconfig [e2e-llm-inference-service] NAME: jaeger [e2e-llm-inference-service] LAST DEPLOYED: Tue Jul 28 01:35:55 2026 [e2e-llm-inference-service] NAMESPACE: observability [e2e-llm-inference-service] STATUS: deployed [e2e-llm-inference-service] REVISION: 1 [e2e-llm-inference-service] TEST SUITE: None [e2e-llm-inference-service] NOTES: [e2e-llm-inference-service] ################################################################### [e2e-llm-inference-service] ### ⚠️ EXPERIMENTAL - NO STABILITY GUARANTEES ### [e2e-llm-inference-service] ### ### [e2e-llm-inference-service] ### This chart is under active development. ### [e2e-llm-inference-service] ### Breaking changes may occur in minor versions. ### [e2e-llm-inference-service] ### ### [e2e-llm-inference-service] ### See README.md for configuration details. ### [e2e-llm-inference-service] ################################################################### [e2e-llm-inference-service] [e2e-llm-inference-service] 🚀 Congratulations on successfully installing Jaeger v2.17.0 (Chart v4.7.0)! [e2e-llm-inference-service] [e2e-llm-inference-service] To access the query UI: [e2e-llm-inference-service] export POD_NAME=$(kubectl get pods --namespace observability -l "app.kubernetes.io/instance=jaeger,app.kubernetes.io/component=all-in-one" -o jsonpath="{.items[0].metadata.name}") [e2e-llm-inference-service] kubectl port-forward --namespace observability $POD_NAME 16686:16686 --address 0.0.0.0 & [e2e-llm-inference-service] Visit http://127.0.0.1:16686/ [e2e-llm-inference-service] [SUCCESS] Successfully installed Jaeger All-in-One via Helm [e2e-llm-inference-service] [INFO] Waiting for pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' to be created... [e2e-llm-inference-service] [INFO] Found 1 pod(s) with label 'app.kubernetes.io/name=jaeger' [e2e-llm-inference-service] [INFO] Waiting for pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' to be ready... [e2e-llm-inference-service] pod/jaeger-6bbd7d4c9d-sr4pb condition met [e2e-llm-inference-service] [SUCCESS] Pods with label 'app.kubernetes.io/name=jaeger' in namespace 'observability' are ready! [e2e-llm-inference-service] [SUCCESS] Jaeger is ready! [e2e-llm-inference-service] ⏳ Verifying Jaeger Service ports (OTLP 4317, Query 16686)... [e2e-llm-inference-service] ✅ Jaeger (Helm) ready — OTLP http://jaeger.observability.svc.cluster.local:4317 [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-ff96d5959-5lpl7" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-ff96d5959-rs8w6 condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/autogluon-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.26 [e2e-llm-inference-service] Kubernetes Version: v1.34.9 [e2e-llm-inference-service] ClusterVersion desired: 4.21.26 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.26 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.2 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.1 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.1 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.2 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.2 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.1 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.1 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.2 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:95b3fcfd6a46997bfc9b2a3a609d19adade819e90aecabd6142db665882198ae [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:62ab60f9c45b7c3b07bf30293f6be92e64bdd759fbf36392332635abe05f9e67 [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:odh-incubating [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:b5c835d640b650f164ba7f36d7008760c88e2974efa65853ee167323c441580c [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:247952ac232dfdb9198891fa9af76488d402df0a9e8dd7866db1e42afc206909 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:247952ac232dfdb9198891fa9af76488d402df0a9e8dd7866db1e42afc206909 [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu and not autoscaling and not tracing [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.122.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'asyncio': '0.23.8', 'anyio': '4.9.0', 'metadata': '3.1.1', 'httpx': '0.30.0', 'cov': '5.0.0', 'json-report': '1.5.0', 'xdist': '3.6.1'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: asyncio-0.23.8, anyio-4.9.0, metadata-3.1.1, httpx-0.30.0, cov-5.0.0, json-report-1.5.0, xdist-3.6.1 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [73 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] 2026-07-28 01:38:22.809 6550 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-28 01:38:22.810 6547 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] 2026-07-28 01:40:55.709 6547 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-07-28 01:40:55.709 6547 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-49b5edd9 [e2e-llm-inference-service] 2026-07-28 01:40:55.739 6547 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 01:40:55.739 6547 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 01:40:55.746 6547 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 01:40:55.746 6547 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-07-28 01:41:05.405 6547 kserve.trace Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-07-28 01:41:05.405 6547 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace e2e-test-gateway-section-name-propagation-94799d44 [e2e-llm-inference-service] 2026-07-28 01:41:05.430 6547 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 01:41:05.430 6547 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 01:41:05.440 6547 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-28 01:41:05.440 6547 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_finalizer_added [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_finalizer_added [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_blocked_when_referenced [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_blocked_when_referenced [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_allowed_when_unreferenced [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_allowed_when_unreferenced [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_unblocked_after_service_deleted [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_config_deletion_unblocked_after_service_deleted [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_prevented_by_webhook [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_prevented_by_webhook [e2e-llm-inference-service] llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_blocked_by_implicit_reference [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_config_deletion.py::test_well_known_config_deletion_blocked_by_implicit_reference [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_cleanup_hpa[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_cleanup_hpa[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] _ test_llm_autoscaling_hpa_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-hpa-deploy", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_hpa_deployment(test_case: TestCase): [e2e-llm-inference-service] """HPA + Deployment: HPA exists with WVA annotations; pods scale up under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:542: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T01:50:04.472347', start_time = 1785203404.4726162 [e2e-llm-inference-service] duration = 900.8220300674438, timestamp_end = '2026-07-28T02:05:05.294656' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f8d16649ee0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T01:50:36Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T01:50:36Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T01:50:39Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T01:50:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T01:51:13Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T01:51:13Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T01:50:43Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T01:51:13Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T01:50:43Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-hpa-de-ec1dce8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-38916baa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-4c186bcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 in namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-deplo-347a3180 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_hpa_deployment] [2026-07-28T01:50:04.005159] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-hpa-deploy', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T01:50:04.018008] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T01:50:04.472226] end - ✅ in 0.454s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T01:50:04.472347] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T01:50:29Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T01:50:29Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T01:50:29Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T01:50:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T01:50:29Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T01:50:29Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T01:50:29Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T01:50:29Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T01:50:29Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T01:50:36Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T01:50:36Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T01:50:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T01:50:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T01:50:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T01:50:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T01:50:29Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T01:50:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T01:50:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T01:50:36Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T01:50:36Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T01:50:39Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T01:50:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T01:50:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T01:50:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T01:50:29Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T01:50:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T01:50:39Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T01:50:36Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T01:50:36Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T01:50:39Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T01:50:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T01:50:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T01:50:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T01:50:43Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T01:50:36Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T01:50:43Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T01:50:36Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T01:50:36Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T01:50:39Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T01:50:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T01:51:13Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T01:51:13Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T01:50:43Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T01:51:13Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T01:50:43Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 3: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T01:38:36Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 4: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T01:38:36Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 5: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T01:38:36Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T01:50:36Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T01:50:36Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T01:50:39Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T01:50:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T01:51:13Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T01:51:13Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T01:50:43Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T01:51:13Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T01:50:43Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T02:05:05.294656] end - ❌ 900.822s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T01:50:36Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T01:50:36Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T01:50:39Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T01:50:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T01:51:13Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T01:51:13Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T01:50:43Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T01:51:13Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T01:50:43Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T02:05:05.294858] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-de-ec1dce8b'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-38916baa'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-4c186bcf'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-deplo-347a3180'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-hpa-deploy still terminating: ['autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv', 'autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc'] [e2e-llm-inference-service] assert not ['autoscale-hpa-deploy-kserve-66746bd8b9-qmqfv', 'autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-hpa-deploy still terminating: ['autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc'] [e2e-llm-inference-service] assert not ['autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T02:05:25.403350] end - ✅ in 20.108s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_hpa_deployment] [2026-07-28T02:05:25.403482] end - ❌ 921.398s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T01:50:36Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T01:50:36Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T01:50:39Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T01:50:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T01:51:13Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T01:51:13Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T01:50:43Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T01:51:13Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T01:50:43Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-deploy-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_keda_deployment[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], pro... {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_keda [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-keda", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-keda-deploy", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_keda_deployment(test_case: TestCase): [e2e-llm-inference-service] """KEDA + Deployment: ScaledObject exists with WVA annotations; no HPA; pods scale up under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:606: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], pro... {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T02:05:26.004802', start_time = 1785204326.0052953 [e2e-llm-inference-service] duration = 900.2643523216248, timestamp_end = '2026-07-28T02:20:26.269651' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f8d15bc1760> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:05:51Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:05:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:06:09Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:06:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:06:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-keda-deployment-b2150d0b with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-keda-d-27e06c40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-da49e827 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-101f2a9d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 in namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-keda-autoscale-keda-dep-1ac84077 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_keda_deployment] [2026-07-28T02:05:25.932804] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-keda'], prompt='KServe is a', service_name='autoscale-keda-deploy', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-keda-deployment-b2150d0b', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T02:05:25.945981] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T02:05:26.004616] end - ✅ in 0.058s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T02:05:26.004802] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:05:38Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:05:38Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:05:38Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:05:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:05:38Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:05:38Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:05:38Z', 'message': 'ScaledObject not yet visible in cache', 'reason': 'ScaledObjectProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:05:38Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:05:38Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:05:51Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:05:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:05:51Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:05:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:06:09Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:06:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:06:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 6: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T01:38:36Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 7: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T01:38:36Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 8: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T01:38:36Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:05:51Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:05:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:06:09Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:06:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:06:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T02:20:26.269651] end - ❌ 900.264s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:05:51Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:05:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:06:09Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:06:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:06:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T02:20:26.269821] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-deploy', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-deployment-b2150d0b', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-d-27e06c40'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-da49e827'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-101f2a9d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-dep-1ac84077'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-keda-deploy still terminating: ['autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl', 'autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4'] [e2e-llm-inference-service] assert not ['autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl', 'autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-keda-deploy still terminating: ['autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4'] [e2e-llm-inference-service] assert not ['autoscale-keda-deploy-kserve-router-scheduler-79d6bcd899-t58v4'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T02:21:01.594637] end - ✅ in 35.324s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_keda_deployment] [2026-07-28T02:21:01.594764] end - ❌ 935.662s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:05:51Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:05:38Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:06:09Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:06:09Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:06:09Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:05:51Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-keda-deployment-b2150d0b (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _____ test_llm_inference_service[router-managed-workload-pd-cpu-model-pvc] _____ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-mod... {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway="router-gateway-2", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-3", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-4", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] after_test=[ [e2e-llm-inference-service] lambda tc: delete_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — clean path: token-producer in inline config [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-tokenizer-kvcache", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-clean-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — migration path: legacy precise-prefix-cache-scorer [e2e-llm-inference-service] # triggers auto-provisioned tokenizer without explicit tokenizer:{} field [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-migration-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name="publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:866: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...del-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T02:09:45.204550', start_time = 1785204585.204813 [e2e-llm-inference-service] duration = 900.2822165489197, timestamp_end = '2026-07-28T02:24:45.487031' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....svc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f7e09379d00> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:10:10Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:12:10Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:09:58Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:10:29Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:10:29Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-inference-service-48639af5 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1959 Created PVC e2e-pvc-model-storage in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2099 Created model download Job e2e-pvc-model-download in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2124 Model download Job e2e-pvc-model-download completed successfully [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 in namespace e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-28T02:09:45.038816] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-model-pvc-router-manage-e8706282', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-inference-service-48639af5', before_test=[ at 0x7f7e0a4f3740>], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T02:09:45.051220] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T02:09:45.204406] end - ✅ in 0.153s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T02:09:45.204550] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-48639af5', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:09:58Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:09:58Z', 'message': 'Inference Pool e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:09:58Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:09:58Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:09:58Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:09:58Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:09:58Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:09:58Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:09:58Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:10:10Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:09:58Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:10:10Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:09:58Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:10:29Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:10:29Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:10:10Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:12:10Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:09:58Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:10:29Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:10:29Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:10:10Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:12:10Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:09:58Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:10:29Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:10:29Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T02:24:45.487031] end - ❌ 900.282s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:10:10Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:12:10Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:09:58Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:10:29Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:10:29Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:903 [router-managed-workload-pd-cpu-model-pvc] ❌ ERROR: Failed to call llm inference service llmisvc-model-pvc-router-manage-e8706282: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:10:10Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:12:10Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:09:58Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:10:29Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:10:29Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:243 # Diagnostics for 'llmisvc-model-pvc-router-manage-e8706282' in 'e2e-test-llm-inference-service-48639af5' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:247 # LLMInferenceService llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:256 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:45Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:09:45Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:09:45Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-latency-predictor: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tokenizer: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:prefill: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T02:12:10Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] resourceVersion: '58350' [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] - name: workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] - name: model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://af9bcd366af794a488e696ab8cf60c29-1770290243.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://af9bcd366af794a488e696ab8cf60c29-1770290243.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://af9bcd366af794a488e696ab8cf60c29-1770290243.us-east-1.elb.amazonaws.com/publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-latency-predictor: kserve-config-llm-scheduler-latency-predictor [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tokenizer: kserve-config-llm-tokenizer [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:10:10Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:10:10Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:12:10Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:10:10Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: PrefillWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:09:58Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:10:10Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:10:29Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:10:29Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:10:10Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://af9bcd366af794a488e696ab8cf60c29-1770290243.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-cpzzq to ip-10-0-138-218.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:30 e2e-test-llm-inference-service-48639af5 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:33 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.133.0.65/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:33 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-138-218.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:33 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-138-218.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:33 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-138-218.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:24 e2e-test-llm-inference-service-48639af5 job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-cpzzq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:42 e2e-test-llm-inference-service-48639af5 job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:24 e2e-test-llm-inference-service-48639af5 persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:24 e2e-test-llm-inference-service-48639af5 ebs.csi.aws.com_aws-ebs-csi-driver-controller-6d797bc958-f489b_ae54b412-74a2-436a-b9df-12de95af31bd Normal Provisioning External provisioner is provisioning volume for claim "e2e-test-llm-inference-service-48639af5/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:24 e2e-test-llm-inference-service-48639af5 persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:27 e2e-test-llm-inference-service-48639af5 ebs.csi.aws.com_aws-ebs-csi-driver-controller-6d797bc958-f489b_ae54b412-74a2-436a-b9df-12de95af31bd Normal ProvisioningSucceeded Successfully provisioned volume pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:55 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs to ip-10-0-138-218.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:58 e2e-test-llm-inference-service-48639af5 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:10:03 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.133.0.66/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:10:03 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-138-218.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:10:04 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-138-218.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:10:04 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-138-218.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:10:04 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-138-218.ec2.internal Normal Pulled Container image "vllm/vllm-openai-cpu:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:10:04 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-138-218.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:10:04 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-138-218.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:11:53 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-138-218.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.66:8001/health": dial tcp 10.133.0.66:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:55 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc to ip-10-0-140-37.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:58 e2e-test-llm-inference-service-48639af5 attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:55 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:55 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:55 e2e-test-llm-inference-service-48639af5 OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:55 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Secret e2e-test-llm-inference-service-48639af5/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:55 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:55 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Role e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:55 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:55 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:55 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Deployment e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:55 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:55 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:55 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /e2e-test-llm-inference-service-132820f03254470cd8a0146008974a5f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:10:10 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:19:56 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Updated Updated v1.Secret e2e-test-llm-inference-service-48639af5/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:10:09 e2e-test-llm-inference-service-48639af5 LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:55 e2e-test-llm-inference-service-48639af5 replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:55 e2e-test-llm-inference-service-48639af5 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-48639af5 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w to ip-10-0-139-81.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:56 e2e-test-llm-inference-service-48639af5 multus Normal AddedInterface Add eth0 [10.134.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:56 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-139-81.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:56 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-139-81.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:09:56 e2e-test-llm-inference-service-48639af5 kubelet/ip-10-0-139-81.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### init-container 'llm-d-routing-sidecar' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 {"level":"info","ts":1785204604.0306325,"msg":"Proxy starting","Built on":"","From Git SHA":"unknown"} [e2e-llm-inference-service] {"level":"info","ts":1785204604.0310776,"msg":"Proxy configuration","config":"{\"Port\":\"8000\",\"KVConnector\":\"nixlv2\",\"ECConnector\":\"\",\"DataParallelSize\":1,\"MaxIdleConnsPerHost\":1024,\"EnablePrefillerSampling\":false,\"PrefillMaxRetries\":0,\"PrefillRetryBackoff\":200000000,\"UseTLSForPrefiller\":true,\"UseTLSForDecoder\":true,\"UseTLSForEncoder\":false,\"InsecureSkipVerifyForPrefiller\":false,\"InsecureSkipVerifyForEncoder\":false,\"InsecureSkipVerifyForDecoder\":false,\"SecureServing\":true,\"CertPath\":\"/var/run/kserve/tls\",\"MooncakeBootstrapPort\":8998,\"EnableSSRFProtection\":true,\"InferencePoolNamespace\":\"e2e-test-llm-inference-service-48639af5\",\"InferencePoolName\":\"llmisvc-model-pvc-router-manage-e8706282-inference-pool\",\"PoolGroup\":\"inference.networking.x-k8s.io\",\"DecodeChunkSize\":0,\"Tracing\":false,\"MoRIIOWriteMode\":false,\"MoRIIODecodeNotifyPort\":61005,\"MoRIIODecodeHandshakePort\":6301,\"MoRIIODecodePodIP\":\"\",\"MoRIIOParallelDispatch\":false,\"MoRIIOPrefillHandshakePort\":6301,\"MoRIIOPrefillNotifyPort\":61005,\"MoRIIOTPSize\":1,\"MoRIIODPSize\":1,\"MoRIIORemoteHosts\":null,\"MoRIIODPSizeLocal\":0,\"MoRIIODecodeHosts\":null,\"DecoderURL\":\"https://localhost:8001\"}"} [e2e-llm-inference-service] {"level":"info","ts":1785204604.0316963,"logger":"allowlist-validator","msg":"starting SSRF protection allowlist validator","namespace":"e2e-test-llm-inference-service-48639af5","poolName":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","gvr":"inference.networking.x-k8s.io/v1alpha2, Resource=inferencepools"} [e2e-llm-inference-service] {"level":"info","ts":1785204604.045974,"logger":"allowlist-validator","msg":"InferencePool added","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785204604.0547931,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":0,"targets":{}} [e2e-llm-inference-service] {"level":"info","ts":1785204604.0548782,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":0,"targets":{}} [e2e-llm-inference-service] {"level":"info","ts":1785204604.1322515,"logger":"allowlist-validator","msg":"allowlist validator started successfully"} [e2e-llm-inference-service] {"level":"info","ts":1785204604.1335342,"logger":"proxy server on port 8000","msg":"server TLS configured"} [e2e-llm-inference-service] {"level":"info","ts":1785204604.1335633,"logger":"proxy server on port 8000","msg":"starting","addr":"[::]:8000"} [e2e-llm-inference-service] {"level":"info","ts":1785204604.5941353,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204605.604205,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204623.8384726,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204634.0466824,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785204634.055109,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204634.0551567,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204664.0476453,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785204664.059127,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204664.0591824,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204694.048626,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785204694.0609365,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204694.0610054,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204723.854407,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204723.8755174,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204724.048926,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785204724.0569236,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204724.0569756,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204754.0498576,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785204754.0600886,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204754.0601418,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204784.0509312,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785204784.062406,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204784.062457,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204814.0511563,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785204814.0600939,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204814.0601523,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204844.0515504,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785204844.0604472,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204844.0604987,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204874.0523527,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785204874.061783,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204874.0618315,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204904.052828,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785204904.0627458,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204904.0627995,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204934.0539262,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785204934.0631676,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204934.0632179,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204964.0546274,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785204964.0636365,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204964.063684,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204994.054697,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785204994.063631,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204994.063687,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205024.05568,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785205024.0645013,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205024.0645564,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205054.0558,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785205054.064923,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205054.064976,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205084.056515,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785205084.0657125,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205084.0657625,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205114.0566838,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785205114.0656521,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205114.0657003,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205144.0571375,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785205144.0671139,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205144.0671577,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205174.0578566,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785205174.0668516,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205174.0669172,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205204.058443,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785205204.0679734,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205204.0680249,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205234.0590801,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785205234.0689406,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205234.0689902,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205264.059537,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785205264.0689602,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205264.0690153,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205294.0604515,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785205294.0711935,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205294.0712411,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205324.0610065,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785205324.0689752,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205324.0690286,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205354.0619361,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785205354.0710385,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205354.0710905,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205384.0627728,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785205384.0721176,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205384.0721672,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205414.0628812,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785205414.0724442,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205414.0725088,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205444.0631928,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785205444.0734031,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205444.0734725,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205474.0640738,"logger":"allowlist-validator","msg":"InferencePool updated","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1785205474.0734785,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785205474.0736036,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.66":{},"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs":{}}} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 [W728 02:11:14.294423989 utils.cpp:76] Warning: numa_migrate_pages failed. errno: 1 (function init_cpu_threads_env) [e2e-llm-inference-service] [W728 02:11:14.294449880 utils.cpp:103] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_threads_env) [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 02:11:14 [v1/worker/cpu_worker.py:109] OMP threads binding of Process 74: [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 02:11:14 [v1/worker/cpu_worker.py:109] OMP tid: 74, core 4 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 02:11:14 [v1/worker/cpu_worker.py:109] OMP tid: 91, core 5 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 02:11:14 [v1/worker/cpu_worker.py:109] OMP tid: 92, core 6 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 02:11:14 [v1/worker/cpu_worker.py:109] OMP tid: 93, core 7 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 02:11:14 [v1/worker/cpu_worker.py:109] [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 02:11:14 [distributed/parallel_state.py:1356] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.133.0.66:34571 backend=gloo [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 02:11:14 [distributed/parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.133.0.66:34571 backend=gloo [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 02:11:14 [distributed/parallel_state.py:1459] Detected 1 nodes in the distributed environment [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 02:11:14 [distributed/parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 02:11:14 [v1/sample/logits_processor/__init__.py:65] No logitsprocs plugins installed (group vllm.logits_processors). [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 02:11:14 [model_executor/offloader/base.py:107] Offloader set to NoopOffloader (no offloading). [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 02:11:14 [v1/worker/cpu_model_runner.py:71] Starting to load model /mnt/models... [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 02:11:14 [compilation/decorators.py:213] Inferred dynamic dimensions for forward method of : ['input_ids', 'positions', 'intermediate_tensors', 'inputs_embeds'] [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 02:11:15 [config/compilation.py:1194] enabled custom ops: Counter() [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 02:11:15 [config/compilation.py:1195] disabled custom ops: Counter({'vocab_parallel_embedding': 1, 'logits_processor': 1}) [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 02:11:15 [model_executor/model_loader/base_loader.py:63] Loading weights on cpu ... [e2e-llm-inference-service] (EngineCore pid=74) Loading pt checkpoint shards: 0% Completed | 0/1 [00:00 [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:24 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:34 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:44 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 02:11:52 [compilation/decorators.py:640] saved AOT compiled function to /home/.cache/vllm/torch_compile_cache/torch_aot_compile/86c9c3c579382eef68a98ac1d59b39811ba08abef3b4e90675a32c8dec3d7c90/rank_0_0/model [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 02:11:53 [compilation/monitor.py:76] Initial profiling/warmup run took 1.05 s [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 02:11:53 [v1/worker/cpu_model_runner.py:92] Warming up done. [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 02:11:53 [v1/engine/core.py:283] init engine (profile, create kv cache, warmup model) took 37.97 seconds [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 02:11:53 [tokenizers/registry.py:68] Loading CachedHfTokenizer for tokenizer_mode='hf' [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:54 [v1/engine/utils.py:1047] Waiting for 1 local, 0 remote core engine proc(s) to start. [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 02:11:54 [utils/gc_utils.py:40] GC Debug Config. enabled:False,top_objects:-1 [e2e-llm-inference-service] (EngineCore pid=74) INFO 07-28 02:11:54 [config/vllm.py:790] Asynchronous scheduling is disabled. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:54 [v1/engine/utils.py:1158] READY from local core engine process 0. [e2e-llm-inference-service] (EngineCore pid=74) WARNING 07-28 02:11:54 [config/vllm.py:859] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 02:11:54 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:54 [v1/metrics/loggers.py:273] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 227 [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 02:11:54 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (EngineCore pid=74) DEBUG 07-28 02:11:54 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:54 [entrypoints/openai/api_server.py:590] Supported tasks: ['generate'] [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:55 [renderers/base.py:197] Warming up chat template processing... [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:55 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:55 [renderers/hf.py:122] Traceback (most recent call last): [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:55 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:55 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools) [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:55 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:55 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:55 [renderers/hf.py:122] raise ValueError( [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:55 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:55 [renderers/hf.py:141] There is no chat template fallback for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [renderers/hf.py:314] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:55 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:55 [renderers/hf.py:122] Traceback (most recent call last): [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:55 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:55 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools) [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:55 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:55 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:55 [renderers/hf.py:122] raise ValueError( [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:55 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:11:55 [renderers/base.py:205] This model does not support chat template. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/openai/api_server.py:594] Starting vLLM server on https://0.0.0.0:8001 [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:37] Available routes are: [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /openapi.json, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /docs, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /redoc, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /tokenize, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /detokenize, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /load, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /version, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /health, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /metrics, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /v1/models, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /ping, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /ping, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /invocations, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /v1/chat/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /v1/responses, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /v1/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /v1/messages, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /inference/v1/generate, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /scale_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /v1/chat/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/launcher.py:46] Route: /v1/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO: Started server process [1] [e2e-llm-inference-service] (APIServer pid=1) INFO: Waiting for application startup. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:11:55 [entrypoints/ssl.py:60] SSLCertRefresher monitors files: ['/var/run/kserve/tls/tls.key', '/var/run/kserve/tls/tls.crt'] [e2e-llm-inference-service] (APIServer pid=1) INFO: Application startup complete. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:12:03 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:12:03 [entrypoints/ssl.py:64] File change detected: deleted - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:12:03 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:12:03 [entrypoints/ssl.py:64] File change detected: modified - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:12:03 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:12:03 [entrypoints/ssl.py:64] File change detected: modified - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:12:03 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:12:03 [entrypoints/ssl.py:64] File change detected: deleted - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:12:03 [entrypoints/ssl.py:34] Reloading SSL certificate chain [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:12:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:12:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:12:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:12:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:12:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:12:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:13:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:13:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:13:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:13:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:13:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:13:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:14:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:14:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:14:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:14:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:14:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:14:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:15:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:15:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:15:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:15:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:15:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:15:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:16:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:16:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:16:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:16:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:16:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:16:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:17:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:17:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:17:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:17:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:17:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:17:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:18:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:18:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:18:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:18:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:18:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:18:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:19:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:19:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:19:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:19:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:19:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:19:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:20:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:20:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:20:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:20:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:20:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:20:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:21:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:21:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:21:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:21:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:21:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:21:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:22:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:22:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:22:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:22:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:22:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:22:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:23:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:23:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:23:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:23:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:23:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:23:55 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:24:05 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:24:15 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:24:25 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:24:35 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-28 02:24:45 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:228 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '7bc245d2-6b1d-49ee-9842-e52be7cd2072', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Tue, 28 Jul 2026 02:24:45 GMT', 'Content-Length': '247'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 {"level":"info","ts":1785204596.5060747,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"unknown","build-ref":""} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5064287,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n- parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef: active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"http","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","pool-namespace":"e2e-test-llm-inference-service-48639af5","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1785204596.506697,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.507974,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: disagg-headers-handler} {Type: prefill-filter} {Type: decode-filter} {Type: queue-scorer} {Type: kv-cache-utilization-scorer} {Type: active-request-scorer} {Type: prefix-cache-scorer} {Type: max-score-picker} {Type: always-disagg-pd-decider} {Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {Type: metrics-data-source, Parameters: {\"scheme\":\"https\"}}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5080159,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5083191,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"llm-d.ai/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"disagg-headers-handler","parameters":null,"type":"disagg-headers-handler"},{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"kv-cache-utilization-scorer","parameters":null,"type":"kv-cache-utilization-scorer"},{"name":"active-request-scorer","parameters":null,"type":"active-request-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"always-disagg-pd-decider","parameters":null,"type":"always-disagg-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"always-disagg-pd-decider"}},"type":"disagg-profile-handler"},{"name":"metrics-data-source","parameters":{"scheme":"https"},"type":"metrics-data-source"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"kv-cache-utilization-scorer","weight":2},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"active-request-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5277061,"caller":"disagg/disagg_profile_handler.go:186","msg":"No deciders.encode configured, E disaggregation disabled"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.527774,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5278285,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.527863,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.52791,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.527974,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5279987,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5280564,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: disagg-headers-handler, Type: disagg-headers-handler} {Name: prefill-filter, Type: prefill-filter} {Name: decode-filter, Type: decode-filter} {Name: queue-scorer, Type: queue-scorer} {Name: kv-cache-utilization-scorer, Type: kv-cache-utilization-scorer} {Name: active-request-scorer, Type: active-request-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: always-disagg-pd-decider, Type: always-disagg-pd-decider} {Name: disagg-profile-handler, Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {Name: metrics-data-source, Type: metrics-data-source, Parameters: {\"scheme\":\"https\"}} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: kv-cache-utilization-scorer, Weight: 2.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: active-request-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5281248,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5281794,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.528209,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"prefix-cache-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5282269,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"disagg-profile-handler"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5282354,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"inflight-load-producer/inflight-load-producer","dataKey":"InFlightLoadDataKey/inflight-load-producer","consumer":"active-request-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5284126,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5284252,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [active-request-scorer/active-request-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [prefix-cache-scorer/prefix-cache-scorer: 3.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc000777300 DataConfig:{Sources:[{Plugin:0xc00049ad80 Extractors:[0xc000777700]}]} FlowControlConfig: ParserRegistry:0xc000777e00}"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5471585,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5471919,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5472074,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5472121,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5472167,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.547221,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5472252,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5472293,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5472336,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5472386,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.547243,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5472476,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5472543,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5472589,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [active-request-scorer/active-request-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [prefix-cache-scorer/prefix-cache-scorer: 3.000000, queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.547292,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5473006,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5473142,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1785204596.547351,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5473626,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.547438,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1785204596.547454,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5474837,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5476718,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5478022,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5479703,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1785204596.54825,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5482934,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5484314,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5484536,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.548588,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5486944,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5533166,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5533183,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.5536227,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.554121,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.6492367,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.6492543,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.6492682,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785204596.649275,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785204596.6492696,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.6492875,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785204596.6492634,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785204596.6493573,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785204596.6494498,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"63c883c3-a17c-48b0-8594-dba63de13e31"} [e2e-llm-inference-service] {"level":"info","ts":1785204609.748886,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"837a417a-d0f5-4824-ad43-3d548aa43ce3"} [e2e-llm-inference-service] {"level":"info","ts":1785204723.8741639,"caller":"inflightload/producer.go:308","msg":"Injected dynamic attribute into endpoint","key":"InFlightLoadDataKey/inflight-load-producer","endpoint":"e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs-rank-0"} [e2e-llm-inference-service] {"level":"info","ts":1785204723.8742065,"caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs","namespace":"e2e-test-llm-inference-service-48639af5"},"namespace":"e2e-test-llm-inference-service-48639af5","name":"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs","reconcileID":"e7453d9c-7913-43fa-846b-53e01c740556"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: e88fc4cd-7159-4057-a77d-a4650be8c05b [e2e-llm-inference-service] resourceVersion: '57266' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T02:10:28Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:10:28Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.41 [e2e-llm-inference-service] nodeName: ip-10-0-139-81.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w [e2e-llm-inference-service] uid: 553127d1-2dd7-4c00-a50f-abdd6b190186 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: f287a026-8b51-4fb8-b3d0-076cde1acc32 [e2e-llm-inference-service] resourceVersion: '58277' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T02:12:03Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.66 [e2e-llm-inference-service] nodeName: ip-10-0-138-218.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs [e2e-llm-inference-service] uid: 958b6fc7-dfbc-4f43-9da6-3bdb212df050 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 958b6fc7-dfbc-4f43-9da6-3bdb212df050 [e2e-llm-inference-service] resourceVersion: '58274' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.66/23"],"mac_address":"0a:58:0a:85:00:42","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.66/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.66\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:42\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c [e2e-llm-inference-service] uid: 16652d6f-f5b2-4123-a695-871fef7149b4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-138-218 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"16652d6f-f5b2-4123-a695-871fef7149b4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:10:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.66"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-c9c6w [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-c9c6w [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000970000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-c9c6w [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000970000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] nodeName: ip-10-0-138-218.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c20 [e2e-llm-inference-service] fsGroup: 1000970000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-ws6bz [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-d6c9z [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:10:04Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:10:04Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:12:03Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:12:03Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] hostIP: 10.0.138.218 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.138.218 [e2e-llm-inference-service] podIP: 10.133.0.66 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.66 [e2e-llm-inference-service] startTime: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-28T02:10:04Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar@sha256:4d57f8fe65a63b44f345487af2b831e64ddf7fb01126d4f51cc8cf49b7412d00 [e2e-llm-inference-service] containerID: cri-o://b59926b6c6a031cda0e5e61e8bf7cd7de8ab9bd666a5e8989c9c0c1735026b05 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-c9c6w [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000970000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000970000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-28T02:10:04Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: docker.io/vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: docker.io/vllm/vllm-openai-cpu@sha256:a8257c201fd2f696c146615987be61579aa2052367b73c27bd3fb1ab8e90bd4b [e2e-llm-inference-service] containerID: cri-o://5190846334f6b35205ccf198245c8c6c5dbebe27060dddb8bb4f20aef5887a70 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-c9c6w [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000970000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000970000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqbjtc [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 568c4ac2-0da4-4b97-8d6f-f9554a808bbb [e2e-llm-inference-service] resourceVersion: '56672' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.58/23"],"mac_address":"0a:58:0a:84:00:3a","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.58/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 [e2e-llm-inference-service] uid: a581b73d-b426-42d5-9cc3-31af19c71974 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-140-37 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a581b73d-b426-42d5-9cc3-31af19c71974"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-4hpkv [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM 0.22.0+\ [e2e-llm-inference-service] \ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" != *\"\ [e2e-llm-inference-service] --kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"* ]] &&\ [e2e-llm-inference-service] \ [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\"\ [e2e-llm-inference-service] \n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"\ [e2e-llm-inference-service] facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-4hpkv [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000970000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-140-37.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c20 [e2e-llm-inference-service] fsGroup: 1000970000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-ws6bz [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] hostIP: 10.0.140.37 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.140.37 [e2e-llm-inference-service] startTime: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-4hpkv [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w [e2e-llm-inference-service] generateName: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 553127d1-2dd7-4c00-a50f-abdd6b190186 [e2e-llm-inference-service] resourceVersion: '57263' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.41/23"],"mac_address":"0a:58:0a:86:00:29","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.41/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.41\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:29\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8 [e2e-llm-inference-service] uid: d200980b-7f0f-4cdf-a594-ec9f167f557e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-139-81 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"d200980b-7f0f-4cdf-a594-ec9f167f557e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:56Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:10:28Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.41"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-gmqqf [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type:\ [e2e-llm-inference-service] \ disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type:\ [e2e-llm-inference-service] \ queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type:\ [e2e-llm-inference-service] \ disagg-profile-handler\n- parameters:\n scheme: https\n type: metrics-data-source\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: queue-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: kv-cache-utilization-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: max-score-picker\n- name: decode\n plugins:\n - pluginRef:\ [e2e-llm-inference-service] \ decode-filter\n - pluginRef: active-request-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-gmqqf [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000970000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-139-81.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c20 [e2e-llm-inference-service] fsGroup: 1000970000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-ws6bz [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-fj8rq [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:09:57Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:10:28Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:10:28Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] hostIP: 10.0.139.81 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.139.81 [e2e-llm-inference-service] podIP: 10.134.0.41 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.41 [e2e-llm-inference-service] startTime: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-28T02:09:56Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:7fd1d61cb5505e026eebab0b9653758779c3dee241e37bea92a0bab961a48c50 [e2e-llm-inference-service] containerID: cri-o://251339ca2820ecbb0b66fcc0fb8d3ab1678779ca464b918dd7123e7379228852 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-gmqqf [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000970000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000970000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 4a5e09ea-44a1-459f-857f-9810366f3cc8 [e2e-llm-inference-service] resourceVersion: '56658' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-fj8rq [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-fj8rq"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-ws6bz"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-ws6bz [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-fj8rq [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-ws6bz [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-fj8rq [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: ff800c95-55d3-4578-bf69-09c943b041d6 [e2e-llm-inference-service] resourceVersion: '56621' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-d6c9z [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-d6c9z"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-ws6bz"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-ws6bz [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-d6c9z [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-ws6bz [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-d6c9z [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: c8a2f595-fd78-4f48-8399-e814e91af9c4 [e2e-llm-inference-service] resourceVersion: '56686' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.46.18 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.46.18 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 2a4c23cf-270b-409a-8c13-c5722c5e7098 [e2e-llm-inference-service] resourceVersion: '56647' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.88.191 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.88.191 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: aab1c9fd-19b8-4d86-afd7-3759b82c4d43 [e2e-llm-inference-service] resourceVersion: '58280' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T02:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T02:12:03Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:12:03Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T02:12:03Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: e19e0ce3-f934-4480-bbb2-421f9354809b [e2e-llm-inference-service] resourceVersion: '63049' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T02:19:56Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T02:19:56Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:19:56Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 4236b87b-d634-4f71-acb5-29a0a92217ea [e2e-llm-inference-service] resourceVersion: '57267' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T02:10:28Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n\ [e2e-llm-inference-service] \ type: disagg-profile-handler\n- parameters:\n scheme: https\n type:\ [e2e-llm-inference-service] \ metrics-data-source\nschedulingProfiles:\n- name: prefill\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n \ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n\ [e2e-llm-inference-service] - name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef:\ [e2e-llm-inference-service] \ active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T02:10:28Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:10:28Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T02:10:28Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 16652d6f-f5b2-4123-a695-871fef7149b4 [e2e-llm-inference-service] resourceVersion: '58279' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] uid: aab1c9fd-19b8-4d86-afd7-3759b82c4d43 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"aab1c9fd-19b8-4d86-afd7-3759b82c4d43"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T02:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-disagg-sidecar:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --enable-tls=decoder [e2e-llm-inference-service] - --enable-tls=prefiller [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b89cd7957 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: a581b73d-b426-42d5-9cc3-31af19c71974 [e2e-llm-inference-service] resourceVersion: '56664' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] uid: e19e0ce3-f934-4480-bbb2-421f9354809b [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e19e0ce3-f934-4480-bbb2-421f9354809b"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 7b89cd7957 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\n# --kv-transfer-config with OffloadingConnector requires vLLM\ [e2e-llm-inference-service] \ 0.22.0+ (vllm-project/vllm#40020).\nKV_TRANSFER_ARGS=\"\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.22.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.22.0\" ]; then\n if [[ \"${VLLM_ADDITIONAL_ARGS:-}\"\ [e2e-llm-inference-service] \ != *\"--kv-transfer-config\"* ]] && [[ \"${VLLM_ADDITIONAL_ARGS:-}\" !=\ [e2e-llm-inference-service] \ *\"--kv_transfer_config\"* ]] && [[ \"$*\" != *\"--kv-transfer-config\"\ [e2e-llm-inference-service] * ]] && [[ \"$*\" != *\"--kv_transfer_config\"* ]]; then\n KV_TRANSFER_ARGS=\"\ [e2e-llm-inference-service] \"\n fi\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n ${KV_TRANSFER_ARGS} \\\n \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-7fc69df7b8 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: d200980b-7f0f-4cdf-a594-ec9f167f557e [e2e-llm-inference-service] resourceVersion: '57265' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] uid: 4236b87b-d634-4f71-acb5-29a0a92217ea [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4236b87b-d634-4f71-acb5-29a0a92217ea"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T02:10:28Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: kv-cache-utilization-scorer\n- type: active-request-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n\ [e2e-llm-inference-service] - parameters:\n deciders:\n prefill: always-disagg-pd-decider\n\ [e2e-llm-inference-service] \ type: disagg-profile-handler\n- parameters:\n scheme: https\n type:\ [e2e-llm-inference-service] \ metrics-data-source\nschedulingProfiles:\n- name: prefill\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: prefill-filter\n - pluginRef: prefix-cache-scorer\n \ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ kv-cache-utilization-scorer\n weight: 2\n - pluginRef: max-score-picker\n\ [e2e-llm-inference-service] - name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef:\ [e2e-llm-inference-service] \ active-request-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 919b0d21-e53d-46f2-9a8a-337041ca03ad [e2e-llm-inference-service] resourceVersion: '56679' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 2f342129-7dda-4db8-9552-89b646d9af98 [e2e-llm-inference-service] resourceVersion: '56627' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 2cc6572c-d9ba-4b64-8199-b634f3c4ea2b [e2e-llm-inference-service] resourceVersion: '56676' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 4c5639d9-f65a-428c-99da-f2e4442f731a [e2e-llm-inference-service] resourceVersion: '56625' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service-z67dd [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-epp-service- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 2e14800d-8b96-480c-bd6c-68e43749862c [e2e-llm-inference-service] resourceVersion: '57264' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T02:10:28Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] uid: c8a2f595-fd78-4f48-8399-e814e91af9c4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T02:10:28Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8a2f595-fd78-4f48-8399-e814e91af9c4"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.41 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w [e2e-llm-inference-service] uid: 553127d1-2dd7-4c00-a50f-abdd6b190186 [e2e-llm-inference-service] nodeName: ip-10-0-139-81.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-s6nq47 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 28892689-17e5-4ce4-afa1-4a3b889c6a48 [e2e-llm-inference-service] resourceVersion: '58276' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T02:12:03Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] uid: 2a4c23cf-270b-409a-8c13-c5722c5e7098 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T02:12:03Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2a4c23cf-270b-409a-8c13-c5722c5e7098"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.66 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs [e2e-llm-inference-service] uid: 958b6fc7-dfbc-4f43-9da6-3bdb212df050 [e2e-llm-inference-service] nodeName: ip-10-0-138-218.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 919b0d21-e53d-46f2-9a8a-337041ca03ad [e2e-llm-inference-service] resourceVersion: '56679' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-48639af5:llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 2f342129-7dda-4db8-9552-89b646d9af98 [e2e-llm-inference-service] resourceVersion: '56627' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-48639af5:llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 2cc6572c-d9ba-4b64-8199-b634f3c4ea2b [e2e-llm-inference-service] resourceVersion: '56676' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] uid: 4c5639d9-f65a-428c-99da-f2e4442f731a [e2e-llm-inference-service] resourceVersion: '56625' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:56Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:10:09Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T02:10:09Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T02:10:10Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] resourceVersion: '57020' [e2e-llm-inference-service] uid: 7e95857e-349d-48ac-8b4d-8afc72a016cb [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:09:56Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:10:09Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:09:57Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:56Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:10:09Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T02:10:09Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T02:10:10Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] resourceVersion: '57020' [e2e-llm-inference-service] uid: 7e95857e-349d-48ac-8b4d-8afc72a016cb [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-48639af5/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:09:56Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:10:09Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:09:57Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T02:10:09Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] resourceVersion: '57001' [e2e-llm-inference-service] uid: 8f8c81aa-9593-4721-9b02-8ef3d363b239 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:10:09Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:10:09Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:58Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:09:58Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T02:09:59Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] resourceVersion: '56868' [e2e-llm-inference-service] uid: 34d4c1bf-ffe5-4c4b-8053-7108779c6d4b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:09:59Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:09:59Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:56Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:09:56Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] resourceVersion: '56717' [e2e-llm-inference-service] uid: e48c5b74-9cf9-420e-ae72-22902d565b22 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:10:09Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:10:09Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] resourceVersion: '57011' [e2e-llm-inference-service] uid: f7a58928-9a1a-4752-8724-76905ff58502 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:56Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:09:56Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] resourceVersion: '56734' [e2e-llm-inference-service] uid: 7e012098-1625-48bc-be7e-28605962b43a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:56Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:09:56Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] resourceVersion: '56717' [e2e-llm-inference-service] uid: e48c5b74-9cf9-420e-ae72-22902d565b22 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:10:09Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:10:09Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] resourceVersion: '57011' [e2e-llm-inference-service] uid: f7a58928-9a1a-4752-8724-76905ff58502 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:56Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:09:56Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] resourceVersion: '56734' [e2e-llm-inference-service] uid: 7e012098-1625-48bc-be7e-28605962b43a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:56Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:09:56Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] resourceVersion: '56717' [e2e-llm-inference-service] uid: e48c5b74-9cf9-420e-ae72-22902d565b22 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:10:09Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:10:09Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] resourceVersion: '57011' [e2e-llm-inference-service] uid: f7a58928-9a1a-4752-8724-76905ff58502 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:56Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:09:56Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] resourceVersion: '56734' [e2e-llm-inference-service] uid: 7e012098-1625-48bc-be7e-28605962b43a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.e2e-test-llm-inference-service-48639af5.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"71b419ee-9abf-4685-8f91-78fac44c2a24"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:09:55Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 71b419ee-9abf-4685-8f91-78fac44c2a24 [e2e-llm-inference-service] resourceVersion: '56698' [e2e-llm-inference-service] uid: bdb3b053-0cba-4d4c-ab9f-18657a91689c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cgjlgs [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:24:46Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 6d9fffb56c [e2e-llm-inference-service] timestamp: '2026-07-28T02:24:25Z' [e2e-llm-inference-service] window: 14.955s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 12823242n [e2e-llm-inference-service] memory: 23756Ki [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 95634503n [e2e-llm-inference-service] memory: 2402544Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-48639af5 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:24:46Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7fc69df7b8 [e2e-llm-inference-service] timestamp: '2026-07-28T02:24:40Z' [e2e-llm-inference-service] window: 18.757s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 55906808n [e2e-llm-inference-service] memory: 33532Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-28T02:24:47.107083] end - ❌ 902.068s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:10:10Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:12:10Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:09:58Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:10:29Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:10:29Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:10:10Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-inference-service-48639af5 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_hpa_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KSer... {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-lws", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-hpa-lws", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_hpa_lws(test_case: TestCase): [e2e-llm-inference-service] """HPA + LWS: HPA exists with WVA annotations; pods scale under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:670: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KSer... {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...ale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T02:21:02.781658', start_time = 1785205262.7819428 [e2e-llm-inference-service] duration = 900.3872640132904, timestamp_end = '2026-07-28T02:36:03.169238' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f8d15bc13a0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:21:47Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:21:27Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:22:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:22:11Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:22:11Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-hpa-lw-1aa98714 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-fe7a55cc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-hpa-b29acdba [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff in namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-hpa-lws-b344a3ff [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_hpa_lws] [2026-07-28T02:21:02.514807] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-hpa-lws', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T02:21:02.527272] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T02:21:02.781528] end - ✅ in 0.254s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T02:21:02.781658] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:21:27Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:21:27Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:21:27Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:21:27Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:21:27Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:21:27Z', 'message': 'HPA not yet visible in cache', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:21:27Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:21:27Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:21:27Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:21:47Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:21:27Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:21:47Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:21:27Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:22:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:22:11Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:22:11Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 9: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T01:38:36Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 10: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T01:38:36Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 11: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T01:38:36Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:21:47Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:21:27Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:22:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:22:11Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:22:11Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T02:36:03.169238] end - ❌ 900.387s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:21:47Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:21:27Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:22:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:22:11Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:22:11Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T02:36:03.169412] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-hpa-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-hpa-lw-1aa98714'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-fe7a55cc'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-hpa-b29acdba'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-hpa-lws-b344a3ff'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 3 pod(s) for autoscale-hpa-lws still terminating: ['autoscale-hpa-lws-kserve-mn-0', 'autoscale-hpa-lws-kserve-mn-0-1', 'autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td'] [e2e-llm-inference-service] assert not ['autoscale-hpa-lws-kserve-mn-0', 'autoscale-hpa-lws-kserve-mn-0-1', 'autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-hpa-lws still terminating: ['autoscale-hpa-lws-kserve-mn-0-1', 'autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td'] [e2e-llm-inference-service] assert not ['autoscale-hpa-lws-kserve-mn-0-1', 'autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-hpa-lws still terminating: ['autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td'] [e2e-llm-inference-service] assert not ['autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-rc5td'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T02:36:38.519584] end - ✅ in 35.350s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_hpa_lws] [2026-07-28T02:36:38.519702] end - ❌ 936.005s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:21:47Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:21:27Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:22:11Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:22:11Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:22:11Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:21:47Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-hpa-lws-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_inference_service[router-managed-workload-simulated-dp-ep-cpu-model-pvc] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-simulated-dp-ep-cpu', 'model-pvc'], prompt='KServe is a', service_name... {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway="router-gateway-1", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-1", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-1", [e2e-llm-inference-service] "router-with-refs-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway="router-gateway-2", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_router_resources( [e2e-llm-inference-service] gateways=[ [e2e-llm-inference-service] make_router_gateway( [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] routes=[ [e2e-llm-inference-service] make_router_main_route( [e2e-llm-inference-service] "router-route-3", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] make_router_health_route( [e2e-llm-inference-service] "router-route-4", [e2e-llm-inference-service] tc.namespace, [e2e-llm-inference-service] "router-gateway-2", [e2e-llm-inference-service] "router-with-refs-pd-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda tc: create_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] after_test=[ [e2e-llm-inference-service] lambda tc: delete_scheduler_configmap(namespace=tc.namespace) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — clean path: token-producer in inline config [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-tokenizer-kvcache", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-clean-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Standalone tokenizer — migration path: legacy precise-prefix-cache-scorer [e2e-llm-inference-service] # triggers auto-provisioned tokenizer without explicit tokenizer:{} field [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="tokenizer-migration-path-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name="publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] "publishers/{namespace}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: "publishers/{namespace}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[lambda tc: ensure_pvc_with_model(namespace=tc.namespace)], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:866: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...pu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T02:25:08.469276', start_time = 1785205508.4695573 [e2e-llm-inference-service] duration = 900.8996031284332, timestamp_end = '2026-07-28T02:40:09.369163' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....p-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f7e08eca2a0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:25:48Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:25:48Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:26:13Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:26:13Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:26:13Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-inference-service-7c3633d1 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1959 Created PVC e2e-pvc-model-storage in namespace e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2099 Created model download Job e2e-pvc-model-download in namespace e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2124 Model download Job e2e-pvc-model-download completed successfully [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-pv-2f105a64 in namespace e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-pv-2f105a64 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-pv-2f105a64 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-699c687c in namespace e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-699c687c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-699c687c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-cfc8d654 in namespace e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-cfc8d654 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-cfc8d654 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-28T02:25:08.103386] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-simulated-dp-ep-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-model-pvc-router-manage-2577e794', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-inference-service-7c3633d1', before_test=[ at 0x7f7e0a4f37e0>], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-2577e794', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-7c3633d1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-2f105a64'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T02:25:08.115854] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-2577e794', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-7c3633d1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-2f105a64'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T02:25:08.469123] end - ✅ in 0.353s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T02:25:08.469276] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-2577e794', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-inference-service-7c3633d1', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-2f105a64'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'Inference Pool e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:25:48Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:25:48Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:25:48Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:25:48Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:25:48Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:25:48Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:25:48Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:26:13Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:26:13Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:26:13Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:25:48Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:25:48Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:26:13Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:26:13Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:26:13Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T02:40:09.369163] end - ❌ 900.900s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:25:48Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:25:48Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:26:13Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:26:13Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:26:13Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:903 [router-managed-workload-simulated-dp-ep-cpu-model-pvc] ❌ ERROR: Failed to call llm inference service llmisvc-model-pvc-router-manage-2577e794: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:25:48Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:25:48Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:26:13Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:26:13Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:26:13Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:243 # Diagnostics for 'llmisvc-model-pvc-router-manage-2577e794' in 'e2e-test-llm-inference-service-7c3633d1' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:244 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:247 # LLMInferenceService llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:256 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:08Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:25:08Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:25:13Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler-latency-predictor: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tokenizer: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T02:26:13Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] resourceVersion: '67933' [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-pv-2f105a64 [e2e-llm-inference-service] - name: workload-simulated-dp-ep-cpu-ll-699c687c [e2e-llm-inference-service] - name: model-pvc-llmisvc-model-pvc-rou-cfc8d654 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://af9bcd366af794a488e696ab8cf60c29-1770290243.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://af9bcd366af794a488e696ab8cf60c29-1770290243.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://af9bcd366af794a488e696ab8cf60c29-1770290243.us-east-1.elb.amazonaws.com/publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler-latency-predictor: kserve-config-llm-scheduler-latency-predictor [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tokenizer: kserve-config-llm-tokenizer [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:25:48Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:25:48Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:25:26Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:26:13Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:26:13Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:26:13Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:25:26Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:25:26Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://af9bcd366af794a488e696ab8cf60c29-1770290243.us-east-1.elb.amazonaws.com/e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:68 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:69 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-7c3633d1 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-qcwtq to ip-10-0-138-218.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:24:52 e2e-test-llm-inference-service-7c3633d1 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-decca439-7a9a-44ad-a2b8-7536bcf29191" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:24:53 e2e-test-llm-inference-service-7c3633d1 multus Normal AddedInterface Add eth0 [10.133.0.67/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:24:53 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-138-218.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6826da04857ec2521ff8d4edc21f858400a7c6db310668e01c2830086507d99d" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:24:53 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-138-218.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:24:53 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-138-218.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:24:47 e2e-test-llm-inference-service-7c3633d1 job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-qcwtq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:03 e2e-test-llm-inference-service-7c3633d1 job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:24:47 e2e-test-llm-inference-service-7c3633d1 persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:24:47 e2e-test-llm-inference-service-7c3633d1 persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:24:47 e2e-test-llm-inference-service-7c3633d1 ebs.csi.aws.com_aws-ebs-csi-driver-controller-6d797bc958-f489b_ae54b412-74a2-436a-b9df-12de95af31bd Normal Provisioning External provisioner is provisioning volume for claim "e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:24:49 e2e-test-llm-inference-service-7c3633d1 ebs.csi.aws.com_aws-ebs-csi-driver-controller-6d797bc958-f489b_ae54b412-74a2-436a-b9df-12de95af31bd Normal ProvisioningSucceeded Successfully provisioned volume pvc-decca439-7a9a-44ad-a2b8-7536bcf29191 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-7c3633d1 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 to ip-10-0-138-218.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:28 e2e-test-llm-inference-service-7c3633d1 attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-decca439-7a9a-44ad-a2b8-7536bcf29191" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:31 e2e-test-llm-inference-service-7c3633d1 multus Normal AddedInterface Add eth0 [10.133.0.68/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:31 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-138-218.ec2.internal Normal Pulled Container image "vllm/vllm-openai-cpu:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:32 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-138-218.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:32 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-138-218.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-7c3633d1 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 to ip-10-0-139-81.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:24 e2e-test-llm-inference-service-7c3633d1 statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:29 e2e-test-llm-inference-service-7c3633d1 attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-decca439-7a9a-44ad-a2b8-7536bcf29191" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:24 e2e-test-llm-inference-service-7c3633d1 leaderworkerset Normal CreatingRevision Creating revision with key 7bc46f896d for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:24 e2e-test-llm-inference-service-7c3633d1 leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:24 e2e-test-llm-inference-service-7c3633d1 leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:24 e2e-test-llm-inference-service-7c3633d1 statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:24 e2e-test-llm-inference-service-7c3633d1 leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:18 e2e-test-llm-inference-service-7c3633d1 OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn: failed to get HTTPRoute e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-2577e794-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:24 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.Secret e2e-test-llm-inference-service-7c3633d1/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:24 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:24 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:24 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:24 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.Service e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:24 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.ServiceAccount e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:24 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /e2e-test-llm-inference-service-5571cf2b7477d60b74b195d7a6a8f03e [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:24 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.Role e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:24 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created Created v1.RoleBinding e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:48 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:35:54 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Updated Updated v1.Secret e2e-test-llm-inference-service-7c3633d1/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:47 e2e-test-llm-inference-service-7c3633d1 LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:24 e2e-test-llm-inference-service-7c3633d1 replicaset-controller Normal SuccessfulCreate Created pod: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:24 e2e-test-llm-inference-service-7c3633d1 deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-66dcb4d65d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 None e2e-test-llm-inference-service-7c3633d1 Normal Scheduled Successfully assigned e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6 to ip-10-0-139-81.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:25 e2e-test-llm-inference-service-7c3633d1 multus Normal AddedInterface Add eth0 [10.134.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:25 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-139-81.ec2.internal Normal Pulled Container image "quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:25 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-139-81.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:80 2026-07-28 02:25:25 e2e-test-llm-inference-service-7c3633d1 kubelet/ip-10-0-139-81.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:228 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'fbe068b1-f0d6-40b8-ba8d-ff52eef6a80b', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Tue, 28 Jul 2026 02:40:09 GMT', 'Content-Length': '236'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:172 ### Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:212 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:225 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:226 INFO 07-28 02:25:43 [importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors. [e2e-llm-inference-service] INFO 07-28 02:25:43 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:25:49 [utils.py:299] [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:25:49 [utils.py:299] █ █ █▄ ▄█ [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:25:49 [utils.py:299] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.19.0 [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:25:49 [utils.py:299] █▄█▀ █ █ █ █ model /mnt/models [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:25:49 [utils.py:299] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:25:49 [utils.py:299] [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:25:49 [utils.py:233] non-default args: {'model_tag': '/mnt/models', 'ssl_keyfile': '/var/run/kserve/tls/tls.key', 'ssl_certfile': '/var/run/kserve/tls/tls.crt', 'enable_ssl_refresh': True, 'model': '/mnt/models', 'served_model_name': ['facebook/opt-125m']} [e2e-llm-inference-service] (APIServer pid=1) WARNING 07-28 02:25:49 [arg_utils.py:1390] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:26:06 [model.py:549] Resolved architecture: OPTForCausalLM [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:26:06 [model.py:1678] Using max model len 2048 [e2e-llm-inference-service] (APIServer pid=1) INFO 07-28 02:26:06 [vllm.py:790] Asynchronous scheduling is enabled. [e2e-llm-inference-service] INFO 07-28 02:26:19 [importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors. [e2e-llm-inference-service] INFO 07-28 02:26:19 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available. [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 02:26:24 [core.py:105] Initializing a V1 LLM engine (v0.19.0) with config: model='/mnt/models', speculative_config=None, tokenizer='/mnt/models', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=facebook/opt-125m, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': None, 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'dce': True, 'nan_asserts': False, 'epilogue_fusion': True, 'cpp.dynamic_threads': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 02:26:24 [cpu_worker.py:236] auto thread-binding list (id, physical core): [(4, 0), (5, 1), (6, 2), (7, 3)] [e2e-llm-inference-service] [W728 02:26:24.427590535 utils.cpp:76] Warning: numa_migrate_pages failed. errno: 1 (function init_cpu_threads_env) [e2e-llm-inference-service] [W728 02:26:24.427619570 utils.cpp:103] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_threads_env) [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 02:26:25 [cpu_worker.py:109] OMP threads binding of Process 41: [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 02:26:25 [cpu_worker.py:109] OMP tid: 41, core 4 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 02:26:25 [cpu_worker.py:109] OMP tid: 58, core 5 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 02:26:25 [cpu_worker.py:109] OMP tid: 59, core 6 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 02:26:25 [cpu_worker.py:109] OMP tid: 60, core 7 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 02:26:25 [cpu_worker.py:109] [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 02:26:25 [parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.133.0.68:45787 backend=gloo [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 02:26:25 [parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-28 02:26:25 [cpu_model_runner.py:71] Starting to load model /mnt/models... [e2e-llm-inference-service] (EngineCore pid=41) Loading pt checkpoint shards: 0% Completed | 0/1 [00:00}, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.304864,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3049228,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.304952,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"no-hit-lru-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.304985,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"approx-prefix-cache-producer"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3050718,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3050818,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000, no-hit-lru-scorer/no-hit-lru-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc000982340 DataConfig:{Sources:[{Plugin:0xc0003898c0 Extractors:[0xc000982540]}]} FlowControlConfig: ParserRegistry:0xc000982a80}"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3222349,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.322257,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3222694,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3222742,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3222785,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3222826,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3222866,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.322291,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.322295,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3223002,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.322304,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3223083,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.322319,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.322324,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, kv-cache-utilization-scorer/kv-cache-utilization-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000, no-hit-lru-scorer/no-hit-lru-scorer: 2.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3223658,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3223734,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3223865,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3223991,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":0} [e2e-llm-inference-service] {"level":"info","ts":1785205525.322411,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.322493,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3225064,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3225296,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3227162,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3228517,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3230083,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3233137,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3234,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3236976,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3238165,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3239985,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3241014,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1785205525.332318,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3344898,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3345468,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.3352387,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"go/pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.4243796,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.4244068,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785205525.4245522,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.4245672,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"llm-d.ai","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785205525.4245846,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.4246073,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"llm-d.ai","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785205525.4246314,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","namespace":"e2e-test-llm-inference-service-7c3633d1"},"namespace":"e2e-test-llm-inference-service-7c3633d1","name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","reconcileID":"94da2139-d4f1-48f6-b8c5-ee0c7481db7e"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.524489,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1785205525.5245175,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1785205547.9882638,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","namespace":"e2e-test-llm-inference-service-7c3633d1"},"namespace":"e2e-test-llm-inference-service-7c3633d1","name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","reconcileID":"1f8bac0d-63b8-4646-adce-1b51d11a2db6"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 95eef377-e6a7-41bb-9e22-a800c90e59ee [e2e-llm-inference-service] resourceVersion: '67687' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T02:25:57Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.44 [e2e-llm-inference-service] nodeName: ip-10-0-139-81.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6 [e2e-llm-inference-service] uid: 7b91a615-cf63-457c-bfcb-74c5828e6f48 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 6028143b-4323-4627-a892-e9c1c020598c [e2e-llm-inference-service] resourceVersion: '67067' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 1579c6d1-9645-4acd-b180-a7e0becfe665 [e2e-llm-inference-service] resourceVersion: '67096' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] apps.kubernetes.io/pod-index: '0' [e2e-llm-inference-service] controller-revision-hash: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-5cf8fb667d [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: ae449cef3b581b6dbab6ee0440d253929da5cb58 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] statefulset.kubernetes.io/pod-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.43/23"],"mac_address":"0a:58:0a:86:00:2b","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.43/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] openshift.io/scc: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: serviceaccount [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] uid: 28c221b8-c4d3-4908-8a6c-7a8d9c7949e3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-139-81 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/size: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:apps.kubernetes.io/pod-index: {} [e2e-llm-inference-service] f:controller-revision-hash: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/worker-index: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:statefulset.kubernetes.io/pod-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"28c221b8-c4d3-4908-8a6c-7a8d9c7949e3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:hostname: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:subdomain: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-ndddm [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: LWS_LEADER_ADDRESS [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0.llmisvc-model-pvc-router-manage-2577e794-kserve-mn.e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] - name: LWS_GROUP_SIZE [e2e-llm-inference-service] value: '2' [e2e-llm-inference-service] - name: LWS_WORKER_INDEX [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-ndddm [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000990000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] nodeName: ip-10-0-139-81.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c30 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-5jrnk [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-m98nw [e2e-llm-inference-service] hostname: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] subdomain: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] hostIP: 10.0.139.81 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.139.81 [e2e-llm-inference-service] startTime: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-ndddm [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: d663462d-861e-4d3e-9cc8-4fa38eda105e [e2e-llm-inference-service] resourceVersion: '67376' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] apps.kubernetes.io/pod-index: '1' [e2e-llm-inference-service] controller-revision-hash: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-74696b7c6f [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: ae449cef3b581b6dbab6ee0440d253929da5cb58 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '1' [e2e-llm-inference-service] statefulset.kubernetes.io/pod-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.68/23"],"mac_address":"0a:58:0a:85:00:44","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.68/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.68\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:44\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/leader-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] openshift.io/scc: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: serviceaccount [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] uid: 4a50b47b-3d92-41c0-8adf-d59feff19456 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-138-218 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/leader-name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/size: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:apps.kubernetes.io/pod-index: {} [e2e-llm-inference-service] f:controller-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-index: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-key: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:statefulset.kubernetes.io/pod-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4a50b47b-3d92-41c0-8adf-d59feff19456"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:hostname: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:subdomain: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:31Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:32Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.68"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-ztlv8 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: LWS_LEADER_ADDRESS [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0.llmisvc-model-pvc-router-manage-2577e794-kserve-mn.e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] - name: LWS_GROUP_SIZE [e2e-llm-inference-service] value: '2' [e2e-llm-inference-service] - name: LWS_WORKER_INDEX [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-ztlv8 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000990000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] nodeName: ip-10-0-138-218.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c30 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-5jrnk [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-m98nw [e2e-llm-inference-service] hostname: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] subdomain: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:25:32Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:25:32Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:25:32Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] hostIP: 10.0.138.218 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.138.218 [e2e-llm-inference-service] podIP: 10.133.0.68 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.68 [e2e-llm-inference-service] startTime: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-28T02:25:32Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: docker.io/vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imageID: docker.io/vllm/vllm-openai-cpu@sha256:a8257c201fd2f696c146615987be61579aa2052367b73c27bd3fb1ab8e90bd4b [e2e-llm-inference-service] containerID: cri-o://91c2d5ed686193244c9a131b811bca5011c1a370a307740456a339e2ff8f77ea [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-ztlv8 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000990000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6 [e2e-llm-inference-service] generateName: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-66dcb4d65d- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 7b91a615-cf63-457c-bfcb-74c5828e6f48 [e2e-llm-inference-service] resourceVersion: '67685' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 66dcb4d65d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.44/23"],"mac_address":"0a:58:0a:86:00:2c","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.44/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.44\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:2c\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-66dcb4d65d [e2e-llm-inference-service] uid: c38aa013-9676-4831-84c8-2f478682c814 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-139-81 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c38aa013-9676-4831-84c8-2f478682c814"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:25Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.44"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-h7zmr [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type:\ [e2e-llm-inference-service] \ single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n -\ [e2e-llm-inference-service] \ pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: no-hit-lru-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-h7zmr [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000990000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-139-81.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c31,c30 [e2e-llm-inference-service] fsGroup: 1000990000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-5jrnk [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-b2mns [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:25:26Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:25:57Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:25:57Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] hostIP: 10.0.139.81 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.139.81 [e2e-llm-inference-service] podIP: 10.134.0.44 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.44 [e2e-llm-inference-service] startTime: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-28T02:25:25Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-llm-d-router-endpoint-picker@sha256:7fd1d61cb5505e026eebab0b9653758779c3dee241e37bea92a0bab961a48c50 [e2e-llm-inference-service] containerID: cri-o://bbacfb1cd95ec144ce5020a460830f005be5e746a70a60fb0eac43a86a5fb230 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-h7zmr [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000990000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000990000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 0d306dfa-c9e2-457d-96b0-bbcaf9ad7625 [e2e-llm-inference-service] resourceVersion: '67075' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-b2mns [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-b2mns"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-5jrnk"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-5jrnk [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-b2mns [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-5jrnk [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-b2mns [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 122f2f91-043e-4c89-8d3b-0a9602ae013a [e2e-llm-inference-service] resourceVersion: '67055' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-m98nw [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-m98nw"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-5jrnk"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: default-dockercfg-5jrnk [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-m98nw [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-5jrnk [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-m98nw [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 9f6915a9-db37-477b-9660-8c4a0fc8f037 [e2e-llm-inference-service] resourceVersion: '67119' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.142.220 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.142.220 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 4244e73e-7b63-4d87-a278-e4eb3da3d940 [e2e-llm-inference-service] resourceVersion: '67065' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.108.149 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.108.149 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-74696b7c6f [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 80ef7761-d4be-43f0-8cef-81a1800a89d9 [e2e-llm-inference-service] resourceVersion: '67092' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] controller.kubernetes.io/hash: 74696b7c6f [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: ae449cef3b581b6dbab6ee0440d253929da5cb58 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] uid: 4a50b47b-3d92-41c0-8adf-d59feff19456 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:data: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:controller.kubernetes.io/hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-index: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-key: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4a50b47b-3d92-41c0-8adf-d59feff19456"}: {} [e2e-llm-inference-service] f:revision: {} [e2e-llm-inference-service] data: [e2e-llm-inference-service] spec: [e2e-llm-inference-service] template: [e2e-llm-inference-service] $patch: replace [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/leader-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: ae449cef3b581b6dbab6ee0440d253929da5cb58 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] revision: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ControllerRevision [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-5cf8fb667d [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 65e4aedd-b84f-40de-bf4f-a624addc87b2 [e2e-llm-inference-service] resourceVersion: '67068' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] controller.kubernetes.io/hash: 5cf8fb667d [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/replicas: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] uid: 28c221b8-c4d3-4908-8a6c-7a8d9c7949e3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:data: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/replicas: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:controller.kubernetes.io/hash: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/worker-index: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"28c221b8-c4d3-4908-8a6c-7a8d9c7949e3"}: {} [e2e-llm-inference-service] f:revision: {} [e2e-llm-inference-service] data: [e2e-llm-inference-service] spec: [e2e-llm-inference-service] template: [e2e-llm-inference-service] $patch: replace [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] revision: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ControllerRevision [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 6cf4a4f5-b215-4d28-9870-375b263c3483 [e2e-llm-inference-service] resourceVersion: '67689' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: no-hit-lru-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T02:25:57Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:25:57Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-28T02:25:57Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-66dcb4d65d" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-66dcb4d65d [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: c38aa013-9676-4831-84c8-2f478682c814 [e2e-llm-inference-service] resourceVersion: '67688' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 66dcb4d65d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler [e2e-llm-inference-service] uid: 6cf4a4f5-b215-4d28-9870-375b263c3483 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6cf4a4f5-b215-4d28-9870-375b263c3483"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 66dcb4d65d [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 66dcb4d65d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.10.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: quay.io/opendatahub/odh-llm-d-router-endpoint-picker:v0.9.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: llm-d.ai/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n-\ [e2e-llm-inference-service] \ type: single-profile-handler\n- type: queue-scorer\n- type: kv-cache-utilization-scorer\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: no-hit-lru-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - parameters:\n scheme: https\n type: metrics-data-source\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: kv-cache-utilization-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: no-hit-lru-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '6' [e2e-llm-inference-service] memory: 16Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 866de71f-9d40-49b6-a3f5-4bcd62e86f1d [e2e-llm-inference-service] resourceVersion: '67113' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: e876b6a7-e3d5-4b67-9767-5a17d097ea8f [e2e-llm-inference-service] resourceVersion: '67051' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-prefill [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: ClusterRole [e2e-llm-inference-service] name: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: df9c0281-bf4e-4383-b25a-cf7cfca3b6df [e2e-llm-inference-service] resourceVersion: '67105' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service-jhzk6 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-epp-service- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 7a08d9f1-809d-4412-b77f-c1ce1e27e923 [e2e-llm-inference-service] resourceVersion: '67686' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T02:25:57Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] uid: 9f6915a9-db37-477b-9660-8c4a0fc8f037 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"9f6915a9-db37-477b-9660-8c4a0fc8f037"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.44 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6 [e2e-llm-inference-service] uid: 7b91a615-cf63-457c-bfcb-74c5828e6f48 [e2e-llm-inference-service] nodeName: ip-10-0-139-81.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-sq2rjj [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc- [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 5c525e16-0984-4f83-9d78-976d1f942f14 [e2e-llm-inference-service] resourceVersion: '67066' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] uid: 4244e73e-7b63-4d87-a278-e4eb3da3d940 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4244e73e-7b63-4d87-a278-e4eb3da3d940"}: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: null [e2e-llm-inference-service] ports: null [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: 866de71f-9d40-49b6-a3f5-4bcd62e86f1d [e2e-llm-inference-service] resourceVersion: '67113' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-7c3633d1:llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: e876b6a7-e3d5-4b67-9767-5a17d097ea8f [e2e-llm-inference-service] resourceVersion: '67051' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-7c3633d1:llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] - system:serviceaccount:e2e-test-llm-inference-service-7c3633d1:llmisvc-model-pvc-router-manage-2577e794-kserve-mn-prefill [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-prefill [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] name: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] uid: df9c0281-bf4e-4383-b25a-cf7cfca3b6df [e2e-llm-inference-service] resourceVersion: '67105' [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - llm-d.ai [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:25:47Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T02:25:48Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T02:25:48Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] resourceVersion: '67565' [e2e-llm-inference-service] uid: c1d6897e-eeb1-4d87-a58f-74334b685b87 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:25:48Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:25:25Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:25:47Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T02:25:48Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T02:25:48Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] resourceVersion: '67565' [e2e-llm-inference-service] uid: c1d6897e-eeb1-4d87-a58f-74334b685b87 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/e2e-test-llm-inference-service-7c3633d1/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:25:48Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:25:25Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T02:25:47Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] resourceVersion: '67541' [e2e-llm-inference-service] uid: 7b88a67e-a688-4d7c-8ff3-fed7d56b0053 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:25:47Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:25:47Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:26Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:25:26Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T02:25:28Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] resourceVersion: '67320' [e2e-llm-inference-service] uid: f3409dfb-ae86-4f40-a9c8-96c6ec3e5264 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:25:28Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:25:28Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] kind: LeaderWorkerSet [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:hpaPodSelector: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] manager: lws [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] - apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderWorkerTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:size: {} [e2e-llm-inference-service] f:workerTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:rolloutStrategy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupPolicy: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] resourceVersion: '67095' [e2e-llm-inference-service] uid: 5f74fee8-251e-49b0-9f2a-ef087ec45ac9 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] leaderWorkerTemplate: [e2e-llm-inference-service] leaderTemplate: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] restartPolicy: RecreateGroupOnPodRestart [e2e-llm-inference-service] size: 2 [e2e-llm-inference-service] workerTemplate: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: vllm/vllm-openai-cpu:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: false [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] networkConfig: [e2e-llm-inference-service] subdomainPolicy: Shared [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] rolloutStrategy: [e2e-llm-inference-service] rollingUpdateConfiguration: [e2e-llm-inference-service] maxSurge: 0 [e2e-llm-inference-service] maxUnavailable: 1 [e2e-llm-inference-service] partition: 0 [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] startupPolicy: LeaderCreated [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] message: Replicas are progressing [e2e-llm-inference-service] reason: GroupsProgressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Progressing [e2e-llm-inference-service] hpaPodSelector: leaderworkerset.sigs.k8s.io/name=llmisvc-model-pvc-router-manage-2577e794-kserve-mn,leaderworkerset.sigs.k8s.io/worker-index=0 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:25Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:25:25Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] resourceVersion: '67147' [e2e-llm-inference-service] uid: 15d17f8c-1d71-48e2-902e-4f4e799d9c1a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-epp-service.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-epp-service.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:48Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:25:48Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] resourceVersion: '67548' [e2e-llm-inference-service] uid: 520180b9-6bef-4223-9067-513694697525 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:25Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:25:25Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] resourceVersion: '67157' [e2e-llm-inference-service] uid: 6c972edb-b667-4e58-93a1-bd679334faa1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:25Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:25:25Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] resourceVersion: '67147' [e2e-llm-inference-service] uid: 15d17f8c-1d71-48e2-902e-4f4e799d9c1a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-epp-service.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-epp-service.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:48Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:25:48Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] resourceVersion: '67548' [e2e-llm-inference-service] uid: 520180b9-6bef-4223-9067-513694697525 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:25Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:25:25Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] resourceVersion: '67157' [e2e-llm-inference-service] uid: 6c972edb-b667-4e58-93a1-bd679334faa1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:25Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:25:25Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] resourceVersion: '67147' [e2e-llm-inference-service] uid: 15d17f8c-1d71-48e2-902e-4f4e799d9c1a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-epp-service.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-epp-service.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:48Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:25:48Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] resourceVersion: '67548' [e2e-llm-inference-service] uid: 520180b9-6bef-4223-9067-513694697525 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:25Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:25:25Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] resourceVersion: '67157' [e2e-llm-inference-service] uid: 6c972edb-b667-4e58-93a1-bd679334faa1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.e2e-test-llm-inference-service-7c3633d1.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"33a24a4e-eb4d-45ad-9b67-a340444d965d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-28T02:25:24Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 33a24a4e-eb4d-45ad-9b67-a340444d965d [e2e-llm-inference-service] resourceVersion: '67131' [e2e-llm-inference-service] uid: 89d9d2bc-1718-4671-8efe-875a3a2db017 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:40:10Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] apps.kubernetes.io/pod-index: '1' [e2e-llm-inference-service] controller-revision-hash: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-74696b7c6f [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: ae449cef3b581b6dbab6ee0440d253929da5cb58 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 7bc46f896d [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '1' [e2e-llm-inference-service] statefulset.kubernetes.io/pod-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] timestamp: '2026-07-28T02:39:46Z' [e2e-llm-inference-service] window: 18.902s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 3718230n [e2e-llm-inference-service] memory: 2411124Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:264 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:265 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6 [e2e-llm-inference-service] namespace: e2e-test-llm-inference-service-7c3633d1 [e2e-llm-inference-service] creationTimestamp: '2026-07-28T02:40:10Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 66dcb4d65d [e2e-llm-inference-service] timestamp: '2026-07-28T02:39:53Z' [e2e-llm-inference-service] window: 19.298s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 570939n [e2e-llm-inference-service] memory: 25296Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-28T02:40:10.454921] end - ❌ 902.351s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:25:48Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:25:48Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:26:13Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:26:13Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:26:13Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:25:26Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-inference-service-7c3633d1 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_keda_lws[router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KSe... {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_keda [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-lws", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-keda", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-keda-lws", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_keda_lws(test_case: TestCase): [e2e-llm-inference-service] """KEDA + LWS: ScaledObject exists with WVA annotations; pods scale under load.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:728: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KSe... {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T02:36:39.969391', start_time = 1785206199.9697394 [e2e-llm-inference-service] duration = 900.4089393615723, timestamp_end = '2026-07-28T02:51:40.378681' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f8d15bc1b20> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:37:11Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-keda-lws-e541a132 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-keda-l-78828c4a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-lws-aut-1696d0b7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-ked-231d315d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 in namespace e2e-test-llm-autoscaling-keda-lws-e541a132 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-keda-autoscale-keda-lws-1337f511 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_keda_lws] [2026-07-28T02:36:39.532196] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-lws', 'prometheus-scrape', 'scaling-keda'], prompt='KServe is a', service_name='autoscale-keda-lws', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-keda-lws-e541a132', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T02:36:39.544853] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T02:36:39.969179] end - ✅ in 0.424s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T02:36:39.969391] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:37:11Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:37:11Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:37:11Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:37:11Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:37:11Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:37:11Z', 'message': 'ScaledObject not yet visible in cache', 'reason': 'ScaledObjectProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:37:11Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:37:11Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:37:11Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:37:11Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 12: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T01:38:36Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 13: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T01:38:36Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 14: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T01:38:36Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:37:11Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T02:51:40.378681] end - ❌ 900.409s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:37:11Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T02:51:40.378759] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-keda-lws', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-keda-lws-e541a132', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-keda-l-78828c4a'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-lws-aut-1696d0b7'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-ked-231d315d'}, [e2e-llm-inference-service] {'name': 'scaling-keda-autoscale-keda-lws-1337f511'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 3 pod(s) for autoscale-keda-lws still terminating: ['autoscale-keda-lws-kserve-mn-0', 'autoscale-keda-lws-kserve-mn-0-1', 'autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd'] [e2e-llm-inference-service] assert not ['autoscale-keda-lws-kserve-mn-0', 'autoscale-keda-lws-kserve-mn-0-1', 'autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-keda-lws still terminating: ['autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd'] [e2e-llm-inference-service] assert not ['autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-rd2vd'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T02:52:25.752018] end - ✅ in 45.373s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_keda_lws] [2026-07-28T02:52:25.752130] end - ❌ 946.220s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:37:11Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'severity': 'Info', 'status': 'True', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:37:54Z', 'message': 'failed to ensure HPA is correctly created for ScaledObject: error parsing prometheus metadata: error parsing prometheus metadata: bearer token= is required when bearer auth is enabled', 'reason': 'ScaledObjectCheckFailed', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-keda-lws-e541a132 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] _ test_llm_autoscaling_cleanup_hpa[router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.autoscaling_hpa [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator-no-replicas", [e2e-llm-inference-service] "prometheus-scrape", [e2e-llm-inference-service] "scaling-hpa", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="autoscale-cleanup-hpa", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_autoscaling_cleanup_hpa(test_case: TestCase): [e2e-llm-inference-service] """Removing scaling config should delete HPA.""" [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] kserve_client = _new_kserve_client() [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] ns = test_case.namespace [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > _create_and_wait(kserve_client, test_case) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:892: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prom... {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] def _create_and_wait(kserve_client, test_case): [e2e-llm-inference-service] """Create LLMISVC and wait for it to be ready.""" [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_autoscaling_wva.py:482: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...e-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-28T02:52:26.868577', start_time = 1785207146.869073 [e2e-llm-inference-service] duration = 900.8776783943176, timestamp_end = '2026-07-28T03:07:27.746754' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....toscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1376: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f8d15bc1d00> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1387: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] all_condition_types = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] ctype = condition.get("type") [e2e-llm-inference-service] all_condition_types.add(ctype) [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(ctype) [e2e-llm-inference-service] [e2e-llm-inference-service] # When TokenizerReady is present, it must also be True [e2e-llm-inference-service] if "TokenizerReady" in all_condition_types: [e2e-llm-inference-service] expected_true_conditions.add("TokenizerReady") [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:53:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1371: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:82 Created test namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret seaweedfs-s3-creds from kserve-ci-e2e-test to e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:131 Copied secret storage-config from kserve-ci-e2e-test to e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:178 Patched default SA in e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 with secret seaweedfs-s3-creds [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:162 ConfigMap odh-kserve-custom-ca-bundle already exists in e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.namespace:namespace.py:159 Copied ConfigMap odh-trusted-ca-bundle from kserve-ci-e2e-test to e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig router-managed-autoscale-cleanu-e5a6b97f in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig router-managed-autoscale-cleanu-e5a6b97f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig router-managed-autoscale-cleanu-e5a6b97f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig workload-llmd-simulator-no-repl-5d16e76d in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-no-repl-5d16e76d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-no-repl-5d16e76d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig prometheus-scrape-autoscale-cle-5a67f5d1 in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig prometheus-scrape-autoscale-cle-5a67f5d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig prometheus-scrape-autoscale-cle-5a67f5d1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1718 Checking LLMInferenceServiceConfig scaling-hpa-autoscale-cleanup-h-aa1ae037 in namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1744 Resource not found, creating LLMInferenceServiceConfig scaling-hpa-autoscale-cleanup-h-aa1ae037 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1754 ✓ Successfully created LLMInferenceServiceConfig scaling-hpa-autoscale-cleanup-h-aa1ae037 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_autoscaling_cleanup_hpa] [2026-07-28T02:52:26.512850] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-llmd-simulator-no-replicas', 'prometheus-scrape', 'scaling-hpa'], prompt='KServe is a', service_name='autoscale-cleanup-hpa', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, namespace='e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1769 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-28T02:52:26.525751] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-28T02:52:26.868418] end - ✅ in 0.342s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-28T02:52:26.868577] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:53:03Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:53:03Z', 'message': 'Inference Pool e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:53:03Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:53:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:53:03Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:53:03Z', 'message': 'The following HTTPRoutes are not ready: [e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:53:03Z', 'message': 'HPA conditions not yet available', 'reason': 'HPAProgressing', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:53:03Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:53:03Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 15: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T01:38:36Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:53:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 16: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T01:38:36Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] INFO common.gateway_proxy_istio:gateway_proxy_istio.py:421 Snapshot 17: restarts={"kserve-ci-e2e-test/router-gateway-1-openshift-default": "error", "kserve-ci-e2e-test/router-gateway-2-openshift-default": "error", "openshift-ingress/openshift-ai-inference-openshift-default": "1\t{\"running\":{\"startedAt\":\"2026-07-28T01:38:36Z\"}}"} resources={"httproutes": 0, "llminferenceservices": 0, "authpolicies": 0} [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1391 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:53:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-28T03:07:27.746754] end - ❌ 900.878s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:53:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [delete_llmisvc] [2026-07-28T03:07:27.746962] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'autoscale-cleanup-hpa', [e2e-llm-inference-service] 'namespace': 'e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-autoscale-cleanu-e5a6b97f'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-no-repl-5d16e76d'}, [e2e-llm-inference-service] {'name': 'prometheus-scrape-autoscale-cle-5a67f5d1'}, [e2e-llm-inference-service] {'name': 'scaling-hpa-autoscale-cleanup-h-aa1ae037'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 2 pod(s) for autoscale-cleanup-hpa still terminating: ['autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8', 'autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq'] [e2e-llm-inference-service] assert not ['autoscale-cleanup-hpa-kserve-6cfdf55fc9-lgnl8', 'autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1394 Waiting: 1 pod(s) for autoscale-cleanup-hpa still terminating: ['autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq'] [e2e-llm-inference-service] assert not ['autoscale-cleanup-hpa-kserve-router-scheduler-6694c4458b-ln5nq'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [delete_llmisvc] [2026-07-28T03:08:03.171353] end - ✅ in 35.424s [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_autoscaling_cleanup_hpa] [2026-07-28T03:08:03.171475] end - ❌ 936.658s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:53:03Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'severity': 'Info', 'status': 'False', 'type': 'ScalingReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-28T02:54:15Z', 'message': 'the HPA was unable to compute the replica count: unable to get external metric e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/wva_desired_replicas/&LabelSelector{MatchLabels:map[string]string{variant_name: autoscale-cleanup-hpa-kserve-hpa,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified', 'reason': 'FailedGetExternalMetric', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ---------------------------- Captured log teardown ----------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.conftest:conftest.py:168 Skipping deletion of namespace e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40 (SKIP_DELETION_ON_FAILURE) [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-utilization-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_flow_control.py::test_flow_control_smoke[cluster_cpu-cluster_single_node-flow-control-concurrency-detector] [e2e-llm-inference-service] llmisvc/test_flow_control.py:47: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-tokenizer-kvcache-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:243: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:92: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] - generated xml file: /workspace/artifacts-dir/junit_e2e-llminferenceservice_and_cluster_cpu_and_not_autoscaling_and_not_tracing.xml - [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results-llminferenceservice_and_cluster_cpu_and_not_autoscaling_and_not_tracing.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_deployment[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_hpa_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_keda_lws[cluster_cpu-cluster_multi_node-router-managed-workload-llmd-simulator-lws-prometheus-scrape-scaling-keda] [e2e-llm-inference-service] FAILED llmisvc/test_llm_autoscaling_wva.py::test_llm_autoscaling_cleanup_hpa[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-no-replicas-prometheus-scrape-scaling-hpa] [e2e-llm-inference-service] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 7 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! [e2e-llm-inference-service] !!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 5 failures !!!!!!!!!!!!! [e2e-llm-inference-service] ====== 7 failed, 42 passed, 3 skipped, 25 warnings in 5385.49s (1:29:45) ======= [must-gather] [must-gather ] OUT 2026-07-28T03:08:06.869965208Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 992373db-dd4b-4aad-8975-5163ff304560 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-28T03:08:06.907457561Z namespace/openshift-must-gather-n226x created [must-gather] [must-gather ] OUT 2026-07-28T03:08:06.917586696Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-59nbw created [must-gather] [must-gather ] OUT 2026-07-28T03:08:06.940335977Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:14.027122620Z [disk usage checker] Started [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:14.030375054Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:14.292431141Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:14.454249280Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:14.454279508Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:14.454279508Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:14.456717229Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:14.621316154Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:14.621350771Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:14.621350771Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:14.623440449Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:14.785743701Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:14.785776062Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:14.785776062Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:14.787798553Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:14.954243491Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:14.954278231Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:14.954278231Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:14.956275772Z Error getting logs from rhods-notebooks [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:15.122121843Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:15.122150366Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:15.122150366Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:15.124275619Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:15.289759580Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:15.289792767Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:15.289792767Z namespaces "istio-system" not found [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:15.291860533Z Error getting logs from istio-system [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:15.449867345Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:15.449896587Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:15.449896587Z namespaces "knative-serving" not found [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:15.451968259Z Error getting logs from knative-serving [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:15.613806941Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:15.613839205Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:15.613839205Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:15.615827542Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:15.900514461Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:16.181403221Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:16.322034456Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:16.407799703Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:16.409781063Z Error collecting info from [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:16.540885101Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:16.632208879Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:16.634111821Z Error collecting info from [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:16.767488998Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:16.851805509Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:16.854518728Z Error collecting info from [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:16.987138365Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:17.068361243Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:17.070386713Z Error collecting info from [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:17.207676772Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:17.295164295Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:17.297414237Z Error collecting info from [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:17.439381680Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:17.523113220Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:17.525197616Z Error collecting info from [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:17.659901692Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:17.742777318Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:17.745064855Z Error collecting info from [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:17.877336754Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:17.962017681Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:17.964511572Z Error collecting info from [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:18.099837719Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:18.182960187Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:18.185097233Z Error collecting info from [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:18.321074038Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:18.409141272Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:18.411677147Z Error collecting info from [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:18.550898537Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:18.632793403Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:18.634884850Z Error collecting info from [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:18.766861049Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:18.855283286Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:18.858616103Z Error collecting info from [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:18.993254951Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:19.034999650Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:19.078037143Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:19.080480460Z Error collecting info from [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:19.213195455Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:19.293328648Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:19.295372047Z Error collecting info from [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:19.431371453Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:19.513232370Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:19.515467974Z Error collecting info from [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:19.651540301Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:19.732466128Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:19.734602549Z Error collecting info from [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:19.871915027Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:19.956520607Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:19.958833213Z Error collecting info from [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:20.096716511Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:20.179189298Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:20.181244883Z Error collecting info from [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:20.885177508Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:21.015734771Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:21.150248631Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:21.282005013Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:21.420322714Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:22.027610572Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:22.165847620Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:22.307067488Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:22.446400916Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:22.575193961Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:23.132324907Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:23.271376420Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:23.358290740Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:23.436354541Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:23.597581845Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:23.734104321Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:23.826905022Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:23.904215222Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:24.039638355Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:24.067712232Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:24.199979200Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:24.273540372Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:24.346970425Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:24.510842333Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:24.645685670Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:24.726087246Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:24.799404160Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:24.964347202Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:25.100394964Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:25.179153119Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:25.253681369Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:25.419017045Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:25.566921893Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:25.641146406Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:25.716698277Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:25.884704726Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:26.027231206Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:26.117029742Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:26.195664281Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:26.364041841Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:26.510452387Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:26.600252549Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:26.682915257Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:26.848183663Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:26.987187634Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:27.077424935Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:27.162580357Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:27.331162857Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:27.473453728Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:27.569767325Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:27.642140855Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:27.808610446Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:27.946289729Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:28.021415324Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:28.094378237Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:28.257908036Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:28.389856904Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:28.469241044Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:28.567780440Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:28.735743215Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:28.882146044Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:28.962198937Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:29.033279900Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:29.044245635Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:29.196497351Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:29.330080262Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:29.404313679Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:29.477380089Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:29.640721439Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:30.197919551Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:30.345720689Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:30.447344473Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:30.614235181Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:31.126259606Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:31.277164951Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:31.384294608Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:31.553227417Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:32.074434848Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:32.231319825Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:32.345854122Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:32.510195620Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:33.029894698Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:33.185937467Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:33.296142200Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:33.464945359Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:34.049116289Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:34.095826856Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:34.282252815Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:34.403845435Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:34.566620723Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:35.432236947Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:35.629800315Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:35.772045570Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:35.948446696Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:36.575313569Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:36.753919828Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:36.874569478Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:37.037411843Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:37.650474876Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:37.827201664Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:37.947825478Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:38.109794114Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:38.710089474Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:38.883010316Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:39.009218063Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:39.053844026Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:39.174408617Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:39.831608597Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:40.006952886Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:40.125834441Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:40.293037744Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:40.906159503Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:41.077417376Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:41.192436227Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:41.354369166Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:41.969846224Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:42.137624840Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:42.251544242Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:42.414461367Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:43.053169353Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:43.231318098Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:43.350497593Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:43.516177433Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:44.060919656Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:44.142791799Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:44.330991774Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:44.453948703Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:44.621613396Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:45.258819860Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:45.434428152Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:45.555402460Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:45.720729313Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:46.331420622Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:46.506206797Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:46.627108334Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:46.801344942Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:47.460940132Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:47.641795557Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:47.769041485Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:47.937399870Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:48.568353039Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:48.746836968Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:48.867976222Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:49.036789690Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:49.065734465Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:49.647900680Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:49.827367384Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:49.963781322Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:50.134909575Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:50.750146635Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:50.925029402Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:51.044543298Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:51.214910072Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:51.835887171Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:52.014417318Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:52.132122034Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:52.298386893Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:52.917051422Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:53.090421665Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:53.208100177Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:53.374919441Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:53.978134912Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:54.072513999Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:54.153005818Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:54.271905076Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:54.438785560Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:55.049772597Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:55.230068475Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:55.349799115Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:55.513080365Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:56.120505149Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:56.294085531Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:56.408479192Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:56.573420940Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:57.191509998Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:57.365626343Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:57.482949019Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:57.655760859Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:58.272696380Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:58.449887299Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:58.568697762Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:58.739617768Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:59.077762364Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:59.396915379Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:59.571802731Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:59.693067955Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:08:59.858165296Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cgszc] POD 2026-07-28T03:09:00.482675700Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cgszc] POD 2026-07-28T03:09:00.657740976Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cgszc] POD 2026-07-28T03:09:00.773471167Z Wrote inspect data to must-gather. [must-gather] [must-gather-cgszc] POD 2026-07-28T03:09:00.792113042Z Caches written to disk [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:01.803102932Z waiting for gather to complete [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:01.806195567Z downloading gather output [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.077388878Z receiving incremental file list [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.090779184Z ./ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.090855355Z aggregated-discovery-api.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.091158832Z aggregated-discovery-apis.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.091908928Z event-filter.html [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.092295126Z timestamp [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.092438389Z version [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.096293621Z cluster-scoped-resources/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.096328232Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.096337602Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.096374593Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.096425054Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.096439704Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.096473355Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.096552107Z namespaces/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.096561637Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.096603858Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.096686019Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.09673676Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.096786602Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.096841353Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.096972616Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.097054627Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.097136669Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.09718243Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.097219921Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.097273092Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.097294123Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.097411125Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.097433535Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.097492597Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.097575649Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.097622919Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.09765581Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.097782553Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.097861384Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.097962097Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.098041888Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.09811423Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.098237073Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.098416576Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.098468077Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.098513998Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.098555199Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.09858621Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.098649812Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.098680452Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.098769514Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.098811295Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.098858926Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.098907377Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.098963918Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.099005679Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.09905614Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.099089651Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.099138892Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.099180313Z namespaces/kuadrant-system/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.099213613Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.099277735Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.099313206Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.099362767Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.099395207Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.099479599Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.099808456Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.100154013Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.100200785Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.100244875Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.100287766Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.100337157Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.100403749Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.10045563Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.100489041Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.100575902Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.100613273Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.100659384Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.100828948Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.100983501Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.101660445Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.101733907Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.102132095Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.102251648Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.102466583Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.102599865Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.102641726Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.102763499Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.10280374Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.102842471Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.102901432Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.102988314Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.103036565Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.103065486Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.103119767Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.103149247Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.103204318Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.103211069Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-4n9wp/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.103255459Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-4n9wp/authorino-7cf4765b85-4n9wp.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.103340151Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-4n9wp/authorino/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.103350001Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-4n9wp/authorino/authorino/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.103355332Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-4n9wp/authorino/authorino/logs/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.103375902Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-4n9wp/authorino/authorino/logs/current.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.118710149Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-4n9wp/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.118782461Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-4n9wp/authorino/authorino/logs/previous.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.118803421Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-vrqn9/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.118868452Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-vrqn9/authorino-operator-56fb6f6856-vrqn9.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.119000345Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-vrqn9/manager/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.119013806Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-vrqn9/manager/manager/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.119020506Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-vrqn9/manager/manager/logs/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.119057636Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-vrqn9/manager/manager/logs/current.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.11924085Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-vrqn9/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.119313462Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-vrqn9/manager/manager/logs/previous.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.119348533Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-5k7td/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.119397134Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-5k7td/dns-operator-controller-manager-ccbdf686-5k7td.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.119490285Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-5k7td/manager/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.119500496Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-5k7td/manager/manager/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.119505516Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-5k7td/manager/manager/logs/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.119530296Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-5k7td/manager/manager/logs/current.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.119643769Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-5k7td/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.11971439Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-5k7td/manager/manager/logs/previous.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.119759621Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-2w4pm/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.119777982Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-2w4pm/kuadrant-console-plugin-598f86468b-2w4pm.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.119871854Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-2w4pm/kuadrant-console-plugin/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.119898704Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-2w4pm/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.119907714Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-2w4pm/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.119943145Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-2w4pm/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.120019397Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-2w4pm/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.120092768Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-2w4pm/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.120128749Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-sfbvm/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.12017162Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-sfbvm/kuadrant-operator-controller-manager-66866f5fcc-sfbvm.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.120303233Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-sfbvm/manager/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.120310233Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-sfbvm/manager/manager/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.120314233Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-sfbvm/manager/manager/logs/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.120354824Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-sfbvm/manager/manager/logs/current.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.138193414Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-sfbvm/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.138264546Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-sfbvm/manager/manager/logs/previous.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.138299297Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-n86fd/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.138341997Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-n86fd/limitador-limitador-85d9db4d5d-n86fd.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.138428549Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-n86fd/limitador/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.138437229Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-n86fd/limitador/limitador/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.138442069Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-n86fd/limitador/limitador/logs/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.13848162Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-n86fd/limitador/limitador/logs/current.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.138577032Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-n86fd/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.138647454Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-n86fd/limitador/limitador/logs/previous.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.138690015Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-vnhf4/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.138720755Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-vnhf4/limitador-operator-controller-manager-774d94476f-vnhf4.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.138813167Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-vnhf4/manager/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.138822948Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-vnhf4/manager/manager/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.138827818Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-vnhf4/manager/manager/logs/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.138836438Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-vnhf4/manager/manager/logs/current.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.139014112Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-vnhf4/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.139084573Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-vnhf4/manager/manager/logs/previous.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.139123724Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.139164725Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.139213766Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.139266567Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.139305758Z namespaces/openshift-ingress/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.139369999Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.139432601Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.139470971Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.139521672Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.139561523Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.139644585Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.13989822Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.140110375Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.140156456Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.140197887Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.140258098Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.140295869Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.140380931Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.140426112Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.140455532Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.140548934Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.140594795Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.140635226Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.141079156Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.141183988Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.141429693Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.141509145Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.14176601Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.141898563Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.142121698Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.142274591Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.142319122Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.142378983Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.142429294Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.142484166Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.142514296Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.142595138Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.142643939Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.14269439Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.142742491Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.142793902Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.142834433Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.142843403Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.142859994Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/istiod-openshift-gateway-5677ccb49b-4r9gw.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.143000457Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/discovery/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.143010137Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/discovery/discovery/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.143014057Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/discovery/discovery/logs/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.143054458Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/discovery/discovery/logs/current.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.164704949Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.164783751Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/discovery/discovery/logs/previous.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.164801361Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.164867303Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.165001226Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/istio-proxy/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.165014256Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/istio-proxy/istio-proxy/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.165019316Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.165060597Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.166415606Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.166511308Z namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.166544359Z namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router-default-68787665fc-c2vr7.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.166659641Z namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.166668091Z namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router/router/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.166673111Z namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router/router/logs/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.166684842Z namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router/router/logs/current.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.166808724Z namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router/router/logs/previous.insecure.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.166897006Z namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router/router/logs/previous.log [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.166930357Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.166979648Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.167030639Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.16707626Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.173781013Z [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.173802923Z sent 2,840 bytes received 1,186,375 bytes 792,810.00 bytes/sec [must-gather] [must-gather-cgszc] OUT 2026-07-28T03:09:02.173807533Z total size is 17,903,659 speedup is 15.06 [must-gather] [must-gather ] OUT 2026-07-28T03:09:02.224237908Z namespace/openshift-must-gather-n226x deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 992373db-dd4b-4aad-8975-5163ff304560 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-28T03:09:02.339340031Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f66492587d98a3cca9bf7ba4541c867770dc76b78b4cf9352e6715dca04502 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 992373db-dd4b-4aad-8975-5163ff304560 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-28T03:09:02.357428896Z namespace/openshift-must-gather-6q4dh created [must-gather] [must-gather ] OUT 2026-07-28T03:09:02.363785832Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-nzwmv created [must-gather] [must-gather ] OUT 2026-07-28T03:09:02.38526749Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f66492587d98a3cca9bf7ba4541c867770dc76b78b4cf9352e6715dca04502 created [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:03.574481091Z [disk usage checker] Started [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:03.578176665Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:03.794217510Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:03.946877502Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:04.040202924Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:04.132819043Z Gathering data for ns/default... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:04.465011502Z Gathering data for ns/openshift... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:04.902876557Z Gathering data for ns/kube-system... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:05.659649838Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.020659244Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.020692066Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.020692066Z namespaces "assisted-installer" not found [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.507273492Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.516360388Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.520633503Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.523349674Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.528588727Z INFO: Gathering HAProxy config files [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.574190110Z Executing Istio gather script [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.576652992Z INFO: Collecting host service logs for crio [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.576976539Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.577327934Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.577684074Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.579065852Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.579356777Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.579709704Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.580057698Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.580414454Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.580771477Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.581150467Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.581603680Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.608303231Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:06.994904957Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:07.044441786Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:07.207826205Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:07.221373316Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:07.506800261Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:07.788263180Z No resources found [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:07.970992884Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:07.971082822Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:08.039505582Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:08.120855504Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-6k54l [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:08.120913024Z insights-runtime-extractor-dzw8g [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:08.120932983Z insights-runtime-extractor-h5pr8 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:08.120951825Z insights-runtime-extractor-mxrqp [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:08.435200271Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:08.460993438Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:08.730883775Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:08.795579804Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:08.869038228Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:08.929775286Z No resources found [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:08.932789429Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:08.960895434Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:09.171783105Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:09.171783105Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:09.179957598Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:09.263351027Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:09.317003364Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:09.409707563Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:09.472044082Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:09.477117981Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:09.572034851Z tar: Removing leading `/' from member names [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:09.578284862Z INFO: HAProxy config collection complete. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:09.654003092Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:09.688287941Z INFO: 'previous-logs' folder not found on ip-10-0-138-218.ec2.internal, skipping... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:09.732400635Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:09.912118265Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:10.092765718Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:10.101857489Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:10.116286252Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:10.134379573Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:10.182226409Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:10.297752154Z INFO: 'previous-logs' folder not found on ip-10-0-139-81.ec2.internal, skipping... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:10.457516544Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:10.635495975Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:10.646596101Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:10.676096945Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:10.912346277Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:10.963742016Z INFO: 'previous-logs' folder not found on ip-10-0-140-37.ec2.internal, skipping... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:10.963848519Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:10.963900171Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:10.994811415Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:11.013896513Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:11.170359647Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:11.175813263Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:11.242358070Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:11.283056239Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:11.345907414Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:11.410333594Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:11.622522355Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:11.626403033Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:11.698359074Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:11.744609510Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:11.795854287Z INFO: INTERCONNECT MODE [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:11.795854287Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:11.797198245Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:11.960412043Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:11.965304242Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:12.007716558Z INFO: Gathering OVN_Northbound from ovnkube-node-ccfz9... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:12.007929963Z INFO: Gathering OVN_Northbound from ovnkube-node-l5mxg... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:12.008160810Z INFO: Gathering OVN_Northbound from ovnkube-node-wrg24... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:12.008433260Z INFO: Gathering OVN_Southbound from ovnkube-node-ccfz9... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:12.009798092Z INFO: Gathering OVN_Southbound from ovnkube-node-l5mxg... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:12.010102924Z INFO: Gathering OVN_Southbound from ovnkube-node-wrg24... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:12.137151766Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:12.229453032Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:12.324925347Z tar: Removing leading `/' from member names [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:12.331003871Z tar: Removing leading `/' from member names [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:12.350683699Z tar: Removing leading `/' from member names [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:12.413622224Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:12.428831496Z tar: Removing leading `/' from member names [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:12.442218465Z tar: Removing leading `/' from member names [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:12.450543017Z tar: Removing leading `/' from member names [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:12.597903384Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:12.658008658Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:12.905302423Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:12.999374566Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:13.134789078Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:13.135856534Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:13.136667345Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:13.138959537Z 7.0M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:13.145905198Z ovnk_database_store/ [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:13.145960313Z ovnk_database_store/ovnkube-node-l5mxg_nbdb [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:13.170578341Z ovnk_database_store/ovnkube-node-l5mxg_sbdb [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:13.218112490Z ovnk_database_store/ovnkube-node-wrg24_sbdb [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:13.276508774Z ovnk_database_store/ovnkube-node-ccfz9_sbdb [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:13.324677869Z ovnk_database_store/ovnkube-node-wrg24_nbdb [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:13.341335792Z ovnk_database_store/ovnkube-node-ccfz9_nbdb [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:13.361790801Z INFO: Network log collection complete. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:13.368462390Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:13.435458947Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:13.696779592Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:13.783098242Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:13.988671348Z W0728 03:09:13.988626 1632 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:13.995892847Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:14.261114195Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:14.361835208Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:14.365198541Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:14.521933246Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:14.779339132Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:14.852787982Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:14.890146071Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:15.019521040Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:15.070013328Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:15.194940504Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:15.260105901Z Gathering data for ns/openshift-console... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:15.326161419Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:15.328986333Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:15.609263596Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:15.856181738Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:15.860369603Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:15.881250988Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:16.047343772Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:16.299628245Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:16.305336355Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:16.364219858Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:16.500498556Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:16.524396403Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:16.755387324Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:16.759172221Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:16.857073087Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:16.971776315Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:17.192346874Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:17.196541317Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:17.404192687Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:17.533577242Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:17.668764293Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:17.671823528Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:17.672974976Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:17.885996295Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:18.095735874Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:18.099854480Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:18.220367307Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:18.357932950Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:18.593235523Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:18.596530128Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:18.634232182Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:18.692645719Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:18.788055116Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:18.788787318Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:18.991520406Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:18.994209561Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:19.187734236Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:19.431098564Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:19.433763612Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:19.486382544Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:19.633347312Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:19.851255657Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:19.872728236Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:19.875842868Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:19.979104882Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:20.063576370Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:20.273023208Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:20.276322136Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:20.469088212Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:20.697748324Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:20.700932561Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:20.898138423Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:20.992487570Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:21.111216191Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:21.114047593Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:21.293423608Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:21.494250818Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:21.496514436Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:21.687201279Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:21.896956623Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:21.899432911Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:22.106218778Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:22.123335514Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:22.325608623Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:22.327961019Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:22.525098203Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:22.739867067Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:22.742263855Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:22.926650013Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:23.034959583Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:23.140172882Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:23.143184600Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:23.301325394Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:23.366513996Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:23.623471294Z Collecting performance related data for node ip-10-0-138-218.ec2.internal [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:23.625825940Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:23.629066614Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:23.631676583Z Collecting performance related data for node ip-10-0-139-81.ec2.internal [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:23.641202029Z Collecting performance related data for node ip-10-0-140-37.ec2.internal [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:23.795272027Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:23.823895224Z Gathering data for ns/kserve... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:23.856348874Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:24.113342193Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:24.279472443Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:24.600907526Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:25.253823533Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:25.460950497Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:25.618803582Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:25.739076884Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:25.949907978Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:26.594426016Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:26.701818439Z Gathering data for ns/openshift-config... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:26.849796756Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:27.034832813Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:27.137289784Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:27.578050836Z Done executing Istio gather script [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:27.664800599Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:27.979911347Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:28.310115644Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:28.703583426Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:28.800992887Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:29.064498011Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:29.386040815Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:29.724673337Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:30.199673519Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:30.542497756Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:31.128230186Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:32.553031557Z Collecting kubelet logs for node ip-10-0-138-218.ec2.internal [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:32.555633891Z Collecting kubelet logs for node ip-10-0-139-81.ec2.internal [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:32.557800345Z Collecting kubelet logs for node ip-10-0-140-37.ec2.internal [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:33.035021085Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:33.044145334Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-6q4dh namespace [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:33.050494431Z INFO: Node performance data collection complete. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:33.807606840Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:34.406362943Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:34.726473323Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:35.287043910Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:35.737276207Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:36.132196461Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:36.434025519Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:36.942863168Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:37.270616076Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:37.686371288Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:37.993999767Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:38.287435097Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:38.814535628Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:38.949300201Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:39.293327628Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:39.670347282Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:40.052124762Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:41.009846507Z Wrote inspect data to must-gather. [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:41.009888882Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:41.009888882Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:41.146758559Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-ng2z5] POD 2026-07-28T03:09:41.395067062Z Caches written to disk [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.403429896Z waiting for gather to complete [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.40644491Z downloading gather output [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.673669937Z receiving incremental file list [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.694201297Z ./ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.694292309Z aggregated-discovery-api.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.694610396Z aggregated-discovery-apis.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.695789391Z event-filter.html [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.698647563Z timestamp [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.698767945Z version [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.706127323Z cluster-scoped-resources/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.706159434Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.706167434Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.706199884Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.706311977Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.706414449Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.706494411Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.706540932Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.706635724Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.706713175Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.706805277Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.706861469Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.707004002Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.707089214Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.707536303Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.707582284Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.707782468Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.708062934Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.708267269Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.708410162Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.712331626Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.713979181Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.714109254Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.714229877Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.714345999Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.714476912Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.715014233Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.715162077Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.71530904Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.715429742Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.716223569Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.716477265Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.716630098Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.716803152Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.718056978Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.718273493Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.718439117Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.71859166Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.718982969Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.719148972Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.719398897Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.719690374Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.720388659Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.720499061Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.720635084Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.720779977Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.72091198Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.721024872Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.721176896Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.721367229Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.722373811Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.722547165Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.722720149Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.724953206Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.725147091Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.725330785Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.72561474Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.726000309Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.72650647Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.726657353Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.726810126Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.726947299Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.727055242Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.727219545Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.727333597Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.727486711Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.727627934Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.727739276Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.728041772Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.728205746Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.728332229Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.728970712Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.729145816Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.729297559Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.729556805Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.730545106Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.730692349Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.730960305Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.731104048Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.731304942Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.731466206Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.73166414Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.731833504Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.732031548Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.732177881Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.732337905Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.732497038Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.732656511Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.732834955Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.73303836Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.733220034Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.733417328Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.733857417Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.734094412Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.734556902Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.734721956Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.735616845Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.735746448Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.73587163Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.736000093Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.736103815Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.73630816Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.736555985Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.736683698Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.73680786Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.736981514Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.llm-d.ai.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.737100627Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.llm-d.ai.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.737219779Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.737380123Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.737508206Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.741484661Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.741981111Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.742565724Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.742740207Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.742903081Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.743044964Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.743184547Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.74329221Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.743400012Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.744232179Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.744627588Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.748481301Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.748621264Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.748778597Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.74892467Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.749055843Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.749168925Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.749282668Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.752298562Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.754655093Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.759833344Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.765150978Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.765305201Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.765608008Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.765768061Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.765931955Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.766044317Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.76619863Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.766488507Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.766594169Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.766700941Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.766903595Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.767031638Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.767154631Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.767269843Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.767380946Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.767514998Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.767655872Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.767767574Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.767874336Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.768020569Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.768179803Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.768282675Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.768434878Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.768546351Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.768673233Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.76898047Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.769097342Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.769362228Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.769490361Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.769612003Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.769714326Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.771699058Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.771828281Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.771957884Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.772068476Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.772174178Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.772315101Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.772431744Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.772590747Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.77271162Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.774396946Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.774546829Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.774680922Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.774832305Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.774969858Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.775147922Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.775388057Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.775898818Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.776169504Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.776285646Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.776398849Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.776532262Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.777158615Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.777358069Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.777465682Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.778996975Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.779148988Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.779299901Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.779397283Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.779567577Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.779844133Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.779975166Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.78019092Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.780575458Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.780677521Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.780797433Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.780981637Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.781142831Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.781276373Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.781432537Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.78156627Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.781900297Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.781974559Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.782069281Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.782156282Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.782240104Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.782313636Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.782465299Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.782565291Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.782650243Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.782739595Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.782823687Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.782931369Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.783019651Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.783105683Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.783197305Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.783280947Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.783368078Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.78345728Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.783544462Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.783632094Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.783721696Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.783804978Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.78390564Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.784043093Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.784132375Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.784243367Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.784328429Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.784413441Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.784511793Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.784593455Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.784679066Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.784769598Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.78485594Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.784966383Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.785051605Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.785138476Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.785221448Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.785348371Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.785438383Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.785529955Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.785614146Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.785703528Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.78578841Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.785891952Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.785982444Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.786066866Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.786155518Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.78625028Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.786363622Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.786456755Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.786549226Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.786637098Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.78672721Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.786822492Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.786926665Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.787038187Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.787137899Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.787239511Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.787338583Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.787436485Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.787527307Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.787619769Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.787717691Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.787806524Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.787896735Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.788034668Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.78812115Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.788211642Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.788302934Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.788390546Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.788484078Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.78858314Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.788665522Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.788756894Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.788847116Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.788953448Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.78905318Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.789143752Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.789231904Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.789329196Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.789430608Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.78951806Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.789611152Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.789702894Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.789793396Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.789899898Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.78999413Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.llm-d.ai.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.790098482Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.790200525Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.790291577Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.790378059Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.79047049Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.790578003Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.790669435Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.790758367Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.790847569Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.790956691Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.791059573Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.791144845Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.791233427Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.791326689Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.791417371Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.791510163Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.791600205Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.791688907Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.791781269Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.79186998Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.791962782Z cluster-scoped-resources/certificates.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.791985213Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.792029104Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-7mwhp.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.792148817Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-gl9rh.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.792249189Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-mnrl4.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.792465993Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.792523045Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.792677948Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.79280376Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.792897002Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.793005345Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.793310961Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.793444184Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.793523406Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.793617138Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.79372652Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.793801522Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.794029927Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.794115729Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.794218901Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.794292603Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.794389365Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.794495777Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.794593569Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.794679471Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.794765633Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.794850815Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.794959117Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.795049719Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.795140341Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.795180071Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.795284644Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.795374636Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.795464077Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.79556259Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.795652201Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.795747023Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.795897727Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.79606544Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.796148722Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.796248554Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.796350256Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.796485379Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.796584792Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.796676754Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.796769756Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.796858387Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.7969776Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.797066092Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.797160174Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.797252836Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.797311567Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.797388779Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.79744025Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.797493511Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.797533802Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.797582063Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.797638984Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.797714746Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.797768227Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.797819078Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.797876469Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.797937621Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.798007402Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.798014562Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.798063393Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.798160465Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.798237587Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.798306109Z cluster-scoped-resources/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.798315009Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.79836135Z cluster-scoped-resources/core/nodes/ip-10-0-138-218.ec2.internal.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.798536293Z cluster-scoped-resources/core/nodes/ip-10-0-139-81.ec2.internal.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.798678786Z cluster-scoped-resources/core/nodes/ip-10-0-140-37.ec2.internal.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.798799399Z cluster-scoped-resources/core/persistentvolumes/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.79884946Z cluster-scoped-resources/core/persistentvolumes/pvc-ca547416-1894-4e76-ab8a-5c73c1bf482a.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.798966092Z cluster-scoped-resources/core/persistentvolumes/pvc-decca439-7a9a-44ad-a2b8-7536bcf29191.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.799025554Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.799097415Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.799135746Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.799237478Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.79930965Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.799414752Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.799496184Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.799576136Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.799726039Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.799817281Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.799968424Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.800067006Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.800152598Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.80024069Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.800330002Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.800418854Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.800516416Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.800601918Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.80068906Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.800784151Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.800871243Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.800982376Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.801070708Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.801140059Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.80119528Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.801275122Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.801376484Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.801460456Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.801546248Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.80162724Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.801773013Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.801896705Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.802008188Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.802065479Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.802077649Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.80208877Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.802166531Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.802175251Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.802214392Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.802279294Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.802315695Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.802395106Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.802403106Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.802449797Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.802543079Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.802693203Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.802733473Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.802743434Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.802797005Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.802899517Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.802911907Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.802950428Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.80302948Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.803076461Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.803187893Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.803252035Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.803315946Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.803358087Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.803427338Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.803473179Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.803582082Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.803633263Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.803696494Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.803728785Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.803785306Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.803828407Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.803875408Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.803920869Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.80398885Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.804088742Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.804154934Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.804201185Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.804257266Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.804304377Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.804362358Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.8044384Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.804502961Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.804543542Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.804644624Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.804655105Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.804699815Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.804775117Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.804814658Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.804944571Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.805052453Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.80536346Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.805466012Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.805582194Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.805697827Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.805786869Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.80584473Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.805902371Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.805992973Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.806075645Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.806217758Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.806347731Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.806434203Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.806531425Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.806618167Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.806700108Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.80678787Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.806876092Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.806989015Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.807068946Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.807154008Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.80723944Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.807326992Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.807414464Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.807503226Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.807586947Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.80768397Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.807759561Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.807898164Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.808022637Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.808121739Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.808207271Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.808297733Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.808442606Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.808544988Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.8086364Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.808726392Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.808824844Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.808932326Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.809026008Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.8091171Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.809208952Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.809298504Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.809397586Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.809486618Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.80958576Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.809673732Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.809761504Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.809856186Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.809968028Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.81006005Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.810142212Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.810239364Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.810325986Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.810410048Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.81050961Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.810591962Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.810686334Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.810785936Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.810830347Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.810844457Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.810896078Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.81096683Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.8109895Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.811006541Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.811099983Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.811109963Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.811141993Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.811188014Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.811235806Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-138-218.ec2.internal.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.811324457Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-139-81.ec2.internal.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.811402499Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-140-37.ec2.internal.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.81145395Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.811488851Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.811578143Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.811641074Z cluster-scoped-resources/storage.k8s.io/volumeattachments/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.811692735Z cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-60a7998b0b9225adb45f1053afefcc4775b1d675bd21632128d1190e277d8b07.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.811776007Z cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-f2a27b43c2e6b90b8d4f30f060b15f737e84b53f40b6fe582cd1bd4a72707980.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.811873749Z host_service_logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.81192735Z host_service_logs/masters/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.811967931Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.812043353Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.812103814Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.812179606Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.812304369Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.81240178Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.812479502Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.812550304Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.812621795Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.812690877Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.812758198Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.81282934Z ingress_controllers/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.81284224Z ingress_controllers/default/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.81284706Z ingress_controllers/default/router-default-68787665fc-c2vr7/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.81285463Z ingress_controllers/default/router-default-68787665fc-c2vr7/haproxy.config [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.813022484Z insights-data/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.813074685Z istio/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.813100366Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.813209918Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.813710919Z istio/event-filter.html [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.813859162Z istio/timestamp [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.813983205Z istio/cluster-scoped-resources/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.813993435Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.813997935Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.814033115Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.814109337Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.814152678Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.814209629Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.81426498Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.814325262Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.814490075Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.815373964Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.815613139Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.815772183Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.816254403Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.816410217Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.816785174Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.817635313Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.817783556Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.817909209Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.818059732Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.818169274Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.818282177Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.81843204Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.818611794Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.81889633Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.819077774Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.819467662Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.819596015Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.819729938Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.819906681Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.819919112Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.819960873Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.820013464Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.820021654Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.820061355Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.820151667Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.82029854Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.820361391Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.820397572Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.820510884Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.820686638Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.82078815Z istio/namespaces/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.82080416Z istio/namespaces/e2e-test-llm-inference-service-48639af5/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.820808601Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.820812691Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.820861652Z istio/namespaces/e2e-test-llm-inference-service-48639af5/gateway.networking.k8s.io/httproutes/llmisvc-model-pvc-router-manage-e8706282-kserve-route.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.821055756Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.821067636Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.821113687Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.821177779Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.821189189Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.821211539Z istio/namespaces/e2e-test-llm-inference-service-48639af5/inference.networking.x-k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.821305421Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.821326032Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.821359532Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.821454265Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.821614528Z istio/namespaces/e2e-test-llm-inference-service-48639af5/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.82169179Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.82170518Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/gateway.networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.82170994Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.821748841Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/gateway.networking.k8s.io/httproutes/llmisvc-model-pvc-router-manage-2577e794-kserve-route.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.821834833Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/inference.networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.821844713Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.821872654Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/inference.networking.k8s.io/inferencepools/llmisvc-model-pvc-router-manage-2577e794-inference-pool.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.821978406Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/inference.networking.x-k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.821991136Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.822024007Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/inference.networking.x-k8s.io/inferencepools/llmisvc-model-pvc-router-manage-2577e794-inference-pool.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.822098528Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/networking.istio.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.822106499Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/networking.istio.io/destinationrules/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.8221552Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.822248182Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.822407045Z istio/namespaces/e2e-test-llm-inference-service-7c3633d1/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.822509807Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.822563838Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.82263388Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.822680341Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.822750792Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.822793873Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.822898785Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.823151061Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.823346745Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.823392596Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.823436567Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.823520899Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.82355959Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.823650272Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.823690273Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.823744794Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.823823735Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.823930608Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.823986379Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.824413058Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.82452318Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.824739105Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.824825387Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.825238416Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.825393849Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.825662255Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.825750137Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.825786727Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.825872069Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.82590119Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.825991432Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.826077214Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.826116494Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.826188006Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.826221177Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.826314519Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.826399091Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.826462502Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.826506253Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.826572784Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.826582824Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.826635756Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.826737318Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.826900741Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.827016604Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.827062215Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.827129666Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.827137366Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.827192317Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/istiod-openshift-gateway-5677ccb49b-4r9gw.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.82730146Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/discovery/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.8273099Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/discovery/discovery/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.82731941Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/discovery/discovery/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.827375192Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/discovery/discovery/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.838195143Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.838265295Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/discovery/discovery/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.838296966Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.838357017Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.838468159Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/istio-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.838487579Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/istio-proxy/istio-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.83849425Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.838547511Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.83989799Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.840057613Z istio/namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.840077454Z istio/namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router-default-68787665fc-c2vr7.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.840207106Z istio/namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.840217997Z istio/namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router/router/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.840223267Z istio/namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router/router/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.840251857Z istio/namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router/router/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.84036043Z istio/namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router/router/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.840431721Z istio/namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router/router/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.840473582Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.840515483Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.840566324Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.840611575Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.840659836Z machine_config_ondisk/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.840669066Z machine_config_termination_logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.840674466Z monitoring/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.840679686Z monitoring/alertmanager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.840701457Z monitoring/alertmanager/status.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.840797749Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.84085599Z monitoring/prometheus/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.840928992Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.841049654Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.841115146Z monitoring/prometheus/rules.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.84177301Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.841813371Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.841851612Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.842841503Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.842901024Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.842993576Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.843094728Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.843218951Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.843306423Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.843361414Z monitoring/prometheus/status/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.843402175Z monitoring/prometheus/status/config.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.843876645Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.843976507Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.844077219Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.844291834Z namespaces/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.844302964Z namespaces/cert-manager-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.844311544Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.844316515Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.844358876Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.844431077Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.844438767Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.844486748Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.844688192Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.844936098Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.845128062Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.845370017Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.845592332Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.845629223Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-8z8b9.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.845791246Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.845817357Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.845892198Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.84596242Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.846035481Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.846053232Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.846152484Z namespaces/cert-manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.846162484Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.846174654Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.846225525Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.846371148Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.846552592Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.846692026Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.84690239Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.847176356Z namespaces/default/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.847218087Z namespaces/default/default.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.847300999Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.847342369Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.847400761Z namespaces/default/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.847441562Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.847541644Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.847795189Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.847870641Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.847978963Z namespaces/default/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.848020464Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.848101186Z namespaces/default/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.848147087Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.848233939Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.848271549Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.84832128Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.848408162Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.848441733Z namespaces/default/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.848496364Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.848587536Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.848686958Z namespaces/default/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.848865022Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.848979415Z namespaces/default/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.849069146Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.849202729Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.849398733Z namespaces/default/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.849463975Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.849504126Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.849570757Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.849607938Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.849648479Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.849681739Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.849775821Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.849854693Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.849927675Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.849970236Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.850047887Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.850078878Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.85015172Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.85016221Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.850216901Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.850374384Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.850547448Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.850692241Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.850905446Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.851205132Z namespaces/default/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.851237163Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.851309654Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.851360075Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.851410027Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.851421157Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.851429357Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.851478648Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.851633741Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.851821415Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.852001479Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.852195263Z namespaces/e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.852373027Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.852380387Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.852390768Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.852457779Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.852613782Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.852814357Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.85297897Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.853179665Z namespaces/e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.853381169Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.853390179Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.853398289Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.85345046Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.853609304Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.853785178Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.853959091Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.854165676Z namespaces/e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.85434871Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.85437117Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.85438643Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.854432361Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.854582964Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.854765439Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.854933992Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.855147897Z namespaces/e2e-test-llm-autoscaling-keda-deployment-b2150d0b/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.855326351Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.855336381Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.855344061Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.855401322Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.855559316Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.855737499Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.855873062Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.856137848Z namespaces/e2e-test-llm-autoscaling-keda-lws-e541a132/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.856331392Z namespaces/e2e-test-llm-inference-service-48639af5/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.856340442Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.856346092Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.856389493Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.856493285Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.856547337Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.856596458Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.85668372Z namespaces/e2e-test-llm-inference-service-48639af5/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.856746461Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.856764611Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.856822703Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.857011967Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.85719346Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.857456036Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.85765086Z namespaces/e2e-test-llm-inference-service-48639af5/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.857828964Z namespaces/e2e-test-llm-inference-service-7c3633d1/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.857839774Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.857845774Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.857896755Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.858003528Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.858053349Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.85809913Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.858185052Z namespaces/e2e-test-llm-inference-service-7c3633d1/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.858249783Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.858289074Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.858336545Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.858485058Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.858663562Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.858799795Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.85904023Z namespaces/e2e-test-llm-inference-service-7c3633d1/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.859223594Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.859232924Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.859242864Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.859297185Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.859442609Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.859620963Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.859755635Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.85998739Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.860218245Z namespaces/kserve/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.860269086Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.860343728Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.860392349Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.86045141Z namespaces/kserve/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.860492871Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.860573263Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.860775977Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.860996762Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.861046263Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.861084664Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.861141365Z namespaces/kserve/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.861183316Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.861262648Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.86137654Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.861429091Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.861511543Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.861553074Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.861563104Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.861601375Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.861686097Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.861773429Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.8618645Z namespaces/kserve/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.861928892Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.862184577Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.86228942Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.862602496Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.862683698Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.862981095Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.863132288Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.863370493Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.863464115Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.863507966Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.863598338Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.863629298Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.86369905Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.863738731Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.863826673Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.863983746Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.864036687Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.864081948Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.864197221Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.864245672Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.864338234Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.864394455Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.864438976Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.864494077Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.864503247Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.864562729Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.864710501Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.864899736Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.865065019Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.865376146Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.86558022Z namespaces/kserve/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.86559492Z namespaces/kserve/pods/kserve-controller-manager-ff96d5959-rs8w6/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.865649522Z namespaces/kserve/pods/kserve-controller-manager-ff96d5959-rs8w6/kserve-controller-manager-ff96d5959-rs8w6.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.865760174Z namespaces/kserve/pods/kserve-controller-manager-ff96d5959-rs8w6/manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.865768844Z namespaces/kserve/pods/kserve-controller-manager-ff96d5959-rs8w6/manager/manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.865782435Z namespaces/kserve/pods/kserve-controller-manager-ff96d5959-rs8w6/manager/manager/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.865820235Z namespaces/kserve/pods/kserve-controller-manager-ff96d5959-rs8w6/manager/manager/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.865972679Z namespaces/kserve/pods/kserve-controller-manager-ff96d5959-rs8w6/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.86605325Z namespaces/kserve/pods/kserve-controller-manager-ff96d5959-rs8w6/manager/manager/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.866106411Z namespaces/kserve/pods/llmisvc-controller-manager-6fcdf65d46-s9w4q/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.866151792Z namespaces/kserve/pods/llmisvc-controller-manager-6fcdf65d46-s9w4q/llmisvc-controller-manager-6fcdf65d46-s9w4q.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.866243574Z namespaces/kserve/pods/llmisvc-controller-manager-6fcdf65d46-s9w4q/manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.866256115Z namespaces/kserve/pods/llmisvc-controller-manager-6fcdf65d46-s9w4q/manager/manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.866268805Z namespaces/kserve/pods/llmisvc-controller-manager-6fcdf65d46-s9w4q/manager/manager/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.866284065Z namespaces/kserve/pods/llmisvc-controller-manager-6fcdf65d46-s9w4q/manager/manager/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.872152421Z namespaces/kserve/pods/llmisvc-controller-manager-6fcdf65d46-s9w4q/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.872222382Z namespaces/kserve/pods/llmisvc-controller-manager-6fcdf65d46-s9w4q/manager/manager/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.872265003Z namespaces/kserve/pods/model-serving-api-7b954ffc95-dj9c7/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.872312575Z namespaces/kserve/pods/model-serving-api-7b954ffc95-dj9c7/model-serving-api-7b954ffc95-dj9c7.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.872409106Z namespaces/kserve/pods/model-serving-api-7b954ffc95-dj9c7/server/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.872421607Z namespaces/kserve/pods/model-serving-api-7b954ffc95-dj9c7/server/server/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.872437207Z namespaces/kserve/pods/model-serving-api-7b954ffc95-dj9c7/server/server/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.872479648Z namespaces/kserve/pods/model-serving-api-7b954ffc95-dj9c7/server/server/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.87256436Z namespaces/kserve/pods/model-serving-api-7b954ffc95-dj9c7/server/server/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.872652962Z namespaces/kserve/pods/model-serving-api-7b954ffc95-dj9c7/server/server/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.872679882Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-b5sgr/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.872724663Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-b5sgr/odh-model-controller-75d9cdbddf-b5sgr.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.872815245Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-b5sgr/manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.872828886Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-b5sgr/manager/manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.872841426Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-b5sgr/manager/manager/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.872894567Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-b5sgr/manager/manager/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.907343915Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-b5sgr/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.907396276Z namespaces/kserve/pods/odh-model-controller-75d9cdbddf-b5sgr/manager/manager/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.907423797Z namespaces/kserve/pods/s3-init-twkpm/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.907474488Z namespaces/kserve/pods/s3-init-twkpm/s3-init-twkpm.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.907602981Z namespaces/kserve/pods/s3-init-twkpm/download-hf-model/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.907612471Z namespaces/kserve/pods/s3-init-twkpm/download-hf-model/download-hf-model/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.907618201Z namespaces/kserve/pods/s3-init-twkpm/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.907636612Z namespaces/kserve/pods/s3-init-twkpm/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.907737574Z namespaces/kserve/pods/s3-init-twkpm/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.907802685Z namespaces/kserve/pods/s3-init-twkpm/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.907836226Z namespaces/kserve/pods/s3-init-twkpm/s3-init/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.907848216Z namespaces/kserve/pods/s3-init-twkpm/s3-init/s3-init/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.907859926Z namespaces/kserve/pods/s3-init-twkpm/s3-init/s3-init/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.907897357Z namespaces/kserve/pods/s3-init-twkpm/s3-init/s3-init/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.908214954Z namespaces/kserve/pods/s3-init-twkpm/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.908283366Z namespaces/kserve/pods/s3-init-twkpm/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.908300946Z namespaces/kserve/pods/seaweedfs-64568bcd49-rtfzg/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.908354647Z namespaces/kserve/pods/seaweedfs-64568bcd49-rtfzg/seaweedfs-64568bcd49-rtfzg.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.908441789Z namespaces/kserve/pods/seaweedfs-64568bcd49-rtfzg/seaweedfs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.908451189Z namespaces/kserve/pods/seaweedfs-64568bcd49-rtfzg/seaweedfs/seaweedfs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.908459429Z namespaces/kserve/pods/seaweedfs-64568bcd49-rtfzg/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.90850288Z namespaces/kserve/pods/seaweedfs-64568bcd49-rtfzg/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.908772336Z namespaces/kserve/pods/seaweedfs-64568bcd49-rtfzg/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.908841967Z namespaces/kserve/pods/seaweedfs-64568bcd49-rtfzg/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.908897888Z namespaces/kserve/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.90894656Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.909001381Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.909053812Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.909145444Z namespaces/kuadrant-system/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.909202255Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.909271067Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.909332228Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.909393639Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.90942922Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.909520272Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.909835419Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.910162166Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.910254738Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.910332579Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.910397501Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.910435621Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.910542764Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.910579035Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.910635836Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.910718507Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.910752978Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.910785309Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.91081817Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.910953243Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.911053675Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.911134796Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.911184008Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.911225598Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.911385162Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.911495824Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.911934484Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.912058266Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.912391493Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.912527826Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.912849903Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.913011877Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.913033717Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.91316487Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.913204481Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.913236472Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.913278782Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.913379424Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.913466506Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.913514587Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.913554638Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.913594799Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.91362394Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.913700582Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.913721722Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.913747632Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.913983668Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.914222133Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.914364526Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.914583581Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.914783965Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.914819535Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-4gld9.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.915169273Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.915220344Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.915317016Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.915393898Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.91548203Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.915538781Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.915557341Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.915621453Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.915667064Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.915770386Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.9159497Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.916054692Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.916153084Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.916162534Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-4n9wp/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.916185755Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-4n9wp/authorino-7cf4765b85-4n9wp.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.916274187Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-4n9wp/authorino/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.916281867Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-4n9wp/authorino/authorino/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.916293427Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-4n9wp/authorino/authorino/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.916349108Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-4n9wp/authorino/authorino/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.929198434Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-4n9wp/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.929275425Z namespaces/kuadrant-system/pods/authorino-7cf4765b85-4n9wp/authorino/authorino/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.929309976Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-vrqn9/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.929384838Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-vrqn9/authorino-operator-56fb6f6856-vrqn9.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.9294974Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-vrqn9/manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.92952049Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-vrqn9/manager/manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.929534481Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-vrqn9/manager/manager/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.929575232Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-vrqn9/manager/manager/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.929755376Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-vrqn9/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.929818547Z namespaces/kuadrant-system/pods/authorino-operator-56fb6f6856-vrqn9/manager/manager/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.929861788Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-5k7td/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.92994164Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-5k7td/dns-operator-controller-manager-ccbdf686-5k7td.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.930061262Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-5k7td/manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.930076182Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-5k7td/manager/manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.930085703Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-5k7td/manager/manager/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.930114873Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-5k7td/manager/manager/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.930239226Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-5k7td/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.930308867Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-ccbdf686-5k7td/manager/manager/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.930352938Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-2w4pm/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.930392879Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-2w4pm/kuadrant-console-plugin-598f86468b-2w4pm.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.930476131Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-2w4pm/kuadrant-console-plugin/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.930483251Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-2w4pm/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.930504722Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-2w4pm/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.930527362Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-2w4pm/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.930620024Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-2w4pm/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.930690386Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-598f86468b-2w4pm/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.930730176Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-sfbvm/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.930759467Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-sfbvm/kuadrant-operator-controller-manager-66866f5fcc-sfbvm.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.93089492Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-sfbvm/manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.93090795Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-sfbvm/manager/manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.930919351Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-sfbvm/manager/manager/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.930988752Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-sfbvm/manager/manager/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.943206904Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-sfbvm/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.943272475Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-66866f5fcc-sfbvm/manager/manager/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.943310536Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-n86fd/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.943369997Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-n86fd/limitador-limitador-85d9db4d5d-n86fd.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.943462309Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-n86fd/limitador/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.943471809Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-n86fd/limitador/limitador/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.94348111Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-n86fd/limitador/limitador/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.943520001Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-n86fd/limitador/limitador/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.943627253Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-n86fd/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.943700745Z namespaces/kuadrant-system/pods/limitador-limitador-85d9db4d5d-n86fd/limitador/limitador/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.943750695Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-vnhf4/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.943770726Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-vnhf4/limitador-operator-controller-manager-774d94476f-vnhf4.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.943901079Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-vnhf4/manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.943917949Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-vnhf4/manager/manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.943923089Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-vnhf4/manager/manager/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.94396586Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-vnhf4/manager/manager/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.944117463Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-vnhf4/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.944196235Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-774d94476f-vnhf4/manager/manager/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.944239896Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.944284357Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.944334488Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.944378149Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.94442941Z namespaces/kube-node-lease/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.94443964Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.94444385Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.944483161Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-138-218.ec2.internal.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.944578593Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-139-81.ec2.internal.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.944663615Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-140-37.ec2.internal.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.944715256Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.944725136Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.944776377Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.945010842Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.945255498Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.945432582Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.945674617Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.945876491Z namespaces/kube-public/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.945906822Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.945914652Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.945988483Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.946140867Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.946322241Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.946470364Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.946672048Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.946952524Z namespaces/kube-system/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.947004205Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.947129528Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.947161959Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.94724064Z namespaces/kube-system/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.947272121Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.947402604Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.947480735Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.947559007Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.947616388Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.947647389Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.947723751Z namespaces/kube-system/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.947756961Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.947844983Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.947893924Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.947963576Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.948052618Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.948092259Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.948100159Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.94815353Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-acgct4ey34r4ng5wr5h7eyriqy.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.948269422Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.948323384Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.948418915Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.948556709Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.94864508Z namespaces/kube-system/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.948676741Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.948844035Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.948961757Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.949243343Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.949328145Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.949605201Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.949741074Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.950107222Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.950161133Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.950205884Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.950293726Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.950373677Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.950438559Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.950471759Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.950567141Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.950686404Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.950735335Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.950792146Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.950829797Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.950902969Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.951023841Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.951034222Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.951067602Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.951246126Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.95144195Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.951588523Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.951979322Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.952232637Z namespaces/kube-system/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.952247268Z namespaces/kube-system/pods/global-pull-secret-syncer-hwr6q/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.952288429Z namespaces/kube-system/pods/global-pull-secret-syncer-hwr6q/global-pull-secret-syncer-hwr6q.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.95237033Z namespaces/kube-system/pods/global-pull-secret-syncer-hwr6q/global-pull-secret-syncer/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.95238088Z namespaces/kube-system/pods/global-pull-secret-syncer-hwr6q/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.952389481Z namespaces/kube-system/pods/global-pull-secret-syncer-hwr6q/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.952441922Z namespaces/kube-system/pods/global-pull-secret-syncer-hwr6q/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.95281392Z namespaces/kube-system/pods/global-pull-secret-syncer-hwr6q/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.952874861Z namespaces/kube-system/pods/global-pull-secret-syncer-hwr6q/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.952969283Z namespaces/kube-system/pods/global-pull-secret-syncer-pqlq7/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.953025064Z namespaces/kube-system/pods/global-pull-secret-syncer-pqlq7/global-pull-secret-syncer-pqlq7.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.953109336Z namespaces/kube-system/pods/global-pull-secret-syncer-pqlq7/global-pull-secret-syncer/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.953121546Z namespaces/kube-system/pods/global-pull-secret-syncer-pqlq7/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.953127076Z namespaces/kube-system/pods/global-pull-secret-syncer-pqlq7/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.953135427Z namespaces/kube-system/pods/global-pull-secret-syncer-pqlq7/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.953527975Z namespaces/kube-system/pods/global-pull-secret-syncer-pqlq7/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.953596706Z namespaces/kube-system/pods/global-pull-secret-syncer-pqlq7/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.953627057Z namespaces/kube-system/pods/global-pull-secret-syncer-wpcdj/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.953660168Z namespaces/kube-system/pods/global-pull-secret-syncer-wpcdj/global-pull-secret-syncer-wpcdj.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.95373967Z namespaces/kube-system/pods/global-pull-secret-syncer-wpcdj/global-pull-secret-syncer/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.95375505Z namespaces/kube-system/pods/global-pull-secret-syncer-wpcdj/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.95376011Z namespaces/kube-system/pods/global-pull-secret-syncer-wpcdj/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.953793181Z namespaces/kube-system/pods/global-pull-secret-syncer-wpcdj/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.954169159Z namespaces/kube-system/pods/global-pull-secret-syncer-wpcdj/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.95424146Z namespaces/kube-system/pods/global-pull-secret-syncer-wpcdj/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.954258521Z namespaces/kube-system/pods/konnectivity-agent-bftn8/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.954313232Z namespaces/kube-system/pods/konnectivity-agent-bftn8/konnectivity-agent-bftn8.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.954393603Z namespaces/kube-system/pods/konnectivity-agent-bftn8/konnectivity-agent/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.954402684Z namespaces/kube-system/pods/konnectivity-agent-bftn8/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.954411224Z namespaces/kube-system/pods/konnectivity-agent-bftn8/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.954453865Z namespaces/kube-system/pods/konnectivity-agent-bftn8/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.955565049Z namespaces/kube-system/pods/konnectivity-agent-bftn8/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.95563737Z namespaces/kube-system/pods/konnectivity-agent-bftn8/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.955678681Z namespaces/kube-system/pods/konnectivity-agent-c9ntv/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.955725122Z namespaces/kube-system/pods/konnectivity-agent-c9ntv/konnectivity-agent-c9ntv.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.955801534Z namespaces/kube-system/pods/konnectivity-agent-c9ntv/konnectivity-agent/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.955808694Z namespaces/kube-system/pods/konnectivity-agent-c9ntv/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.955816894Z namespaces/kube-system/pods/konnectivity-agent-c9ntv/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.955868935Z namespaces/kube-system/pods/konnectivity-agent-c9ntv/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.95700635Z namespaces/kube-system/pods/konnectivity-agent-c9ntv/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.957088631Z namespaces/kube-system/pods/konnectivity-agent-c9ntv/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.957141633Z namespaces/kube-system/pods/konnectivity-agent-wthds/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.957174173Z namespaces/kube-system/pods/konnectivity-agent-wthds/konnectivity-agent-wthds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.957266155Z namespaces/kube-system/pods/konnectivity-agent-wthds/konnectivity-agent/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.957275235Z namespaces/kube-system/pods/konnectivity-agent-wthds/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.957280776Z namespaces/kube-system/pods/konnectivity-agent-wthds/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.957289936Z namespaces/kube-system/pods/konnectivity-agent-wthds/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.95844171Z namespaces/kube-system/pods/konnectivity-agent-wthds/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.958507162Z namespaces/kube-system/pods/konnectivity-agent-wthds/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.958528082Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.958574993Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.958668605Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal/haproxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.958677066Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.958687436Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.958733427Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.959045703Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.959123505Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-218.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.959157116Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.959210647Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.959288888Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal/haproxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.959299479Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.959304299Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.959319449Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.959562324Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.959627546Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.959669537Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.959701447Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.959788919Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal/haproxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.959796179Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.959802839Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.959848311Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.960452884Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.960528915Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-37.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.960550455Z namespaces/kube-system/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.960604917Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.960659858Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.960669808Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.960703929Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.960800641Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.960851722Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.960899413Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.960982055Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.961035006Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.961080547Z namespaces/observability/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.961088567Z namespaces/observability/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.961097427Z namespaces/observability/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.961143648Z namespaces/observability/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.961404894Z namespaces/observability/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.961662219Z namespaces/observability/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.961803232Z namespaces/observability/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.962073278Z namespaces/observability/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.962295063Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.962303953Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.962313613Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.962354064Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.962449466Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.962504407Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.962520878Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.962572879Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.962725572Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.962917866Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.963069409Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.963262454Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.963447458Z namespaces/open-cluster-management-bbfed7ef-61c3-4711-ab5f-0e769310c/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.963458438Z namespaces/open-cluster-management-bbfed7ef-61c3-4711-ab5f-0e769310c/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.963468798Z namespaces/open-cluster-management-bbfed7ef-61c3-4711-ab5f-0e769310c/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.963519609Z namespaces/open-cluster-management-bbfed7ef-61c3-4711-ab5f-0e769310c/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.963669393Z namespaces/open-cluster-management-bbfed7ef-61c3-4711-ab5f-0e769310c/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.963857597Z namespaces/open-cluster-management-bbfed7ef-61c3-4711-ab5f-0e769310c/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.96403926Z namespaces/open-cluster-management-bbfed7ef-61c3-4711-ab5f-0e769310c/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.964251825Z namespaces/open-cluster-management-bbfed7ef-61c3-4711-ab5f-0e769310c/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.964470039Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.964522481Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.964737325Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.964785196Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.964856008Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.964899889Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.965001551Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.965098763Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.965180325Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.965218846Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.965258076Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.965300937Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.965352218Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.965446151Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.965479091Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.965509332Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.965604164Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.965647925Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.965699346Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.966670037Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.966753519Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.96683714Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.966945983Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.967074365Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.967201578Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.967378562Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.967442503Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.967483264Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.967540895Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.967592977Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.967630787Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.967673278Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.96776227Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.967847712Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.967909763Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.968062767Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.968129428Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.968161489Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.96823814Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.968255841Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.968324802Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.968339773Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.968372043Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.968523686Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.968871454Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.969050338Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.969239082Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.969431216Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.969464667Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.969513438Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.969561909Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.969654501Z namespaces/openshift-apiserver/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.969708882Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.969796614Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.969838445Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.969895996Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.969951687Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.970049089Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.970128081Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.970208942Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.970252803Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.970295904Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.970351606Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.970394926Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.970482078Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.97054007Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.97058267Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.970663182Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.970733344Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.970766714Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.970899717Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.97099597Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.971074381Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.971156483Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.971237255Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.971356007Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.971534001Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.971551591Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.971610243Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.971665704Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.971719585Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.971759766Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.971805817Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.971901179Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.972018851Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.972242596Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.972288007Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.972348148Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.972400349Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.972459511Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.972490831Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.972542132Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.972708026Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.97289768Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.973069014Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.973271758Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.973483193Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.973510733Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.973577845Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.973621606Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.973661867Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.973671247Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.973675317Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.973757338Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.973790479Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.973802219Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.973856691Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.974046165Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.974231369Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.974363402Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.974587286Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.974794651Z namespaces/openshift-authentication/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.974804441Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.974812291Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.974897923Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.975056777Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.97523567Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.975371633Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.975576418Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.975745791Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.975758532Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.975762662Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.975795542Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.975864744Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.975873944Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.976012757Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.97616783Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.976346144Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.976481857Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.976697132Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.976898876Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.976935627Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.976943957Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.976993788Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.977050869Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.977064569Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.97710263Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.977167422Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.977176712Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.977222033Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.977409467Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.977710453Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.977852676Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.978099042Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.978322336Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.978361727Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.978426059Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.978458839Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.978533991Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.978564232Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.978653274Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.978742345Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.978813097Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.978862598Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.9789389Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.97897445Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.979019211Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.979100073Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.979144154Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.979175785Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.979256196Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.979305608Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.979314428Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.979364089Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.979452071Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.979492272Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.979597104Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.979673375Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.979753757Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.979830559Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.979924171Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.980127115Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.980234468Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.980281678Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.980319239Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.9803738Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.980411121Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.980457122Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.980505223Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.980573595Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.980622886Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.980700898Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.980782479Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.98082642Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.980853861Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.980935283Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.980989884Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.981039565Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.981046625Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.981083136Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.981243569Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.981439393Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.981578986Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.981802221Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.982020346Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.982051386Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.982110057Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.982161469Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.98220729Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.982264831Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.982301542Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.982355853Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.982387414Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.982475696Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.982519076Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.982588628Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.982618719Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.98271169Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.982757902Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.982893825Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.983013197Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.983094929Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.98314123Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.983188031Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.983238262Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.983285133Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.983361424Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.983420176Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.983452296Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.983542828Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.983602469Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.98361136Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.983650421Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.983738552Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.983820444Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.984033519Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.984293924Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.984341415Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.984444247Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.984519349Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.984993079Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.985079481Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.985300766Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.985458249Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.985573642Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.985621023Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.985662884Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.985712865Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.985744915Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.985800267Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.985831027Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.985902309Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.98594142Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.986045312Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.986126614Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.986175905Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.986202055Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.986257606Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.986297907Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.986369419Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.986378239Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.98643415Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.986583064Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.986761217Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.986909831Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.987137155Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.98734084Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.98735134Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.987397141Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/aws-ebs-csi-driver-node-bmx5t.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.987506923Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/csi-driver/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.987521233Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/csi-driver/csi-driver/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.987529054Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/csi-driver/csi-driver/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.987540744Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.987642916Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.987714688Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.987734768Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/csi-liveness-probe/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.987739808Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.987748149Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.98780201Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.987906032Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.988001144Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.988050885Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/csi-node-driver-registrar/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.988058565Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.988064965Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.988110606Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.988210128Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.98827856Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-bmx5t/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.988327091Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.988356162Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/aws-ebs-csi-driver-node-gtb98.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.988455944Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/csi-driver/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.988471484Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/csi-driver/csi-driver/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.988479534Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/csi-driver/csi-driver/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.988523495Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.988630027Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.988698389Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.98873807Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/csi-liveness-probe/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.98874831Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.98875613Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.988788841Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.988900833Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.988986815Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.989031596Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/csi-node-driver-registrar/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.989039916Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.989046156Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.989096697Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.989185099Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.989253991Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gtb98/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.989310742Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.989365993Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/aws-ebs-csi-driver-node-wnths.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.989447945Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/csi-driver/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.989461865Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/csi-driver/csi-driver/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.989469025Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/csi-driver/csi-driver/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.989515086Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.989608628Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.9896796Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.989736001Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/csi-liveness-probe/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.989750291Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.989760252Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.989792682Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.989896755Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.989978216Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.990115749Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/csi-node-driver-registrar/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.990124479Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.99013102Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.990174921Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.990271652Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.990339234Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wnths/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.990390155Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.990429116Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.990486437Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.990493377Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.990541998Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.99060331Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.990647801Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.990701732Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.990744483Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.990806354Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.990813674Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.990828364Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.990860935Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.990959837Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.990970127Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.991015008Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.99107399Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.99108579Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.991135441Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.991283144Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.991519549Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.991658452Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.991847106Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.992193204Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.992242685Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.992364427Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.992408138Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.99248353Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.992525901Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.992648323Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.992726005Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.992807707Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.993382359Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.99343057Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.993520142Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.993562533Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.993610244Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.993650195Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.993734697Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.993777848Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.993826529Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.993931851Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.993979302Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.993994452Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.994042223Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.994133695Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.994178556Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.995075386Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.995185588Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.995330421Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.995414053Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.995623777Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.99577104Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.995936604Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.996018776Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.996068157Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.996182239Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.99621307Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.996274401Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.996321232Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.996381763Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.996416464Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.996504786Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.996613938Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.99667571Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.996715161Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.996795492Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.996843673Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.996946046Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.997007987Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.997053308Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.997093149Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.99714556Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.99716326Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.997218641Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.997434246Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.99762861Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.997780733Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.998040369Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.998250863Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.998259654Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-bh7l6/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.998326615Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-bh7l6/tuned-bh7l6.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.998433358Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-bh7l6/tuned/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.998443728Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-bh7l6/tuned/tuned/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.998451878Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-bh7l6/tuned/tuned/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.998487768Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-bh7l6/tuned/tuned/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.998607831Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-bh7l6/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.998676993Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-bh7l6/tuned/tuned/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.998746574Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-kmchd/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.998783975Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-kmchd/tuned-kmchd.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.998866947Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-kmchd/tuned/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.998874957Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-kmchd/tuned/tuned/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.998896627Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-kmchd/tuned/tuned/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.998949178Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-kmchd/tuned/tuned/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.999064061Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-kmchd/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.999137933Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-kmchd/tuned/tuned/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.999172043Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-qlbrg/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.999223004Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-qlbrg/tuned-qlbrg.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.999304096Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-qlbrg/tuned/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.999311766Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-qlbrg/tuned/tuned/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.999316466Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-qlbrg/tuned/tuned/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.999347257Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-qlbrg/tuned/tuned/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.999461569Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-qlbrg/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.999532581Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-qlbrg/tuned/tuned/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.999560302Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.999591512Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.999643733Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.999674444Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.999727965Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.999743375Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.999793007Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-138-218.ec2.internal.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:42.999906819Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-139-81.ec2.internal.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.000079643Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-140-37.ec2.internal.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.000127224Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.000176435Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.000289047Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.000337198Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.00040681Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.000450131Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.000510702Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.000546603Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.000635994Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.000744837Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.000846639Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.000926991Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.000985412Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.001031513Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.001077094Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.001162506Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.001219767Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.001253598Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.00136011Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.001431772Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.001467452Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.001652346Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.001730288Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.001858021Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.001974983Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.002102276Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.002220589Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.002397642Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.002452073Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.002497165Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.002558676Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.002599817Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.002680709Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.00272979Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.002801961Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.002912034Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.002962015Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.003015176Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.003064477Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.003108388Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.003176309Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.00322574Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.003253471Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.003303422Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.003349673Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.003360893Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.003392484Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.003540237Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.003799132Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.003964446Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.00417144Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.004366514Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.004375005Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-2gt7p/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.004416046Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-2gt7p/cluster-samples-operator-6748489656-2gt7p.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.004502868Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-2gt7p/cluster-samples-operator-watch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.004512838Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-2gt7p/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.004519518Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-2gt7p/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.004562389Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-2gt7p/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.004660041Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-2gt7p/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.004724692Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-2gt7p/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.004771603Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-2gt7p/cluster-samples-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.004779783Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-2gt7p/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.004790754Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-2gt7p/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.004818214Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-2gt7p/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.005583901Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-2gt7p/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.005649072Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-6748489656-2gt7p/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.005705783Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.005743544Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.005804295Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.005845256Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.005968519Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.00602761Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.006094342Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.006135523Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.006202044Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.006246145Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.006326867Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.006427619Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.006539691Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.006592122Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.006623643Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.006689364Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.006734165Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.006813877Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.006857258Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.006921679Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.007019102Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.007069592Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.007079383Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.007125644Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.007214916Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.007291837Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.007338108Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.007455731Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.007536602Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.00786923Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.008081234Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.008177406Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.00832385Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.008463462Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.008530584Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.008567555Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.008631536Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.008670907Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.008736028Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.008775839Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.00883913Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.008897672Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.009000694Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.009083426Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.009145427Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.009189148Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.009252829Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.00928369Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.009397012Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.009424283Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.009489524Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.009525155Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.009606437Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.009635487Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.009689379Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.009848212Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.010064147Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.01020215Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.010414884Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.010596528Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.010603558Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-xblm8/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.010631339Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-xblm8/volume-data-source-validator-549c6d54cf-xblm8.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.01071485Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-xblm8/volume-data-source-validator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.010721961Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-xblm8/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.010734811Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-xblm8/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.010759682Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-xblm8/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.010860244Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-xblm8/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.010956836Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-549c6d54cf-xblm8/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.010989527Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.011025517Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.011074118Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.011081098Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.011117539Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.0111801Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.011225952Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.011291303Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.011326724Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.011421326Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.011470397Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.011537158Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.011566209Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.01161907Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.011669811Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.011738543Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.011823154Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.011902816Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.011990978Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.012038129Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.01210072Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.012149491Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.012228203Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.012287964Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.012328725Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.012409637Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.012462198Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.012469068Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.012516049Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.012587661Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.012632842Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.012729424Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.012816106Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.012936658Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.01302729Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.013111962Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.013232765Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.013415668Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.01348426Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.013529011Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.013591322Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.013623563Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.013678454Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.013716665Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.013797027Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.013875938Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.01395097Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.014009831Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.014066173Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.014110173Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.014147234Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.014162775Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.014219856Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.014368559Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.014619644Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.014755987Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.014987192Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.015181026Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.015213107Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.015280528Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.015322519Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.015416771Z namespaces/openshift-config-managed/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.015466672Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.015532154Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.015568385Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.015639146Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.015685667Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.015765529Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.015850231Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.016094856Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.016146557Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.016191718Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.016238069Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.016264969Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.016351411Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.016410223Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.016448064Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.016532075Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.016608267Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.016648328Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.019785375Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.019866597Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.02001225Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.020091552Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.020177373Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.020313216Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.020425129Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.02048741Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.020538581Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.020651034Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.020700925Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.020747416Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.020795757Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.020835387Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.020896559Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.020976401Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.021063722Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.021148714Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.021197935Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.021262757Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.021332628Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.021379029Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.021448381Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.021466231Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.021519052Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.021716766Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.021964212Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.022119955Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.022313409Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.022522314Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.022561064Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.022612545Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.022622636Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.022669057Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.022727778Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.022766619Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.02283373Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.022871841Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.022954043Z namespaces/openshift-config-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.022974733Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.022983064Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.023025794Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.023097246Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.023106476Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.023151987Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.02330362Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.023490194Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.023626597Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.023827322Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.024354113Z namespaces/openshift-config/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.024403954Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.024461865Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.024505386Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.024576468Z namespaces/openshift-config/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.024602418Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.02469437Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.024777882Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.024854144Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.024926805Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.024990897Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.025052228Z namespaces/openshift-config/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.025101879Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.025177151Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.025214231Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.025255952Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.025351284Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.025397055Z namespaces/openshift-config/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.025428926Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.025559799Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.02563383Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.025714482Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.025789314Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.025872066Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.02607001Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.026216913Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.026265994Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.026304835Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.026354896Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.026396677Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.026451538Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.026497329Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.02655548Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.026601421Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.026677793Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.026757394Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.026800965Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.026843906Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.026931628Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.026975309Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.027041991Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.027051071Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.027098922Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.027246015Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.027650204Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.028270557Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.028290067Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.028388759Z namespaces/openshift-config/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.028528832Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.028740987Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.028762247Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029055714Z namespaces/openshift-console-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029076974Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029168836Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029497663Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029512813Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029518334Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029522624Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029530354Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029534214Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029537864Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029541904Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029545944Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029549664Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029553174Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029556834Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029560674Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029564335Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029567965Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029571615Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029588515Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029599305Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.029605486Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.030244069Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.030326661Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.030562516Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.030637528Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.03076875Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.030893883Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.031125368Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.031187709Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.03123394Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.031290162Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.031344213Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.031404604Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.031457445Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.031535007Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.031611429Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.031683Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.031719361Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.031779772Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.031827693Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.031905985Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.032041528Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.032094839Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.03214108Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.032184251Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.032196301Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.032276143Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.032430446Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.0326079Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.032745833Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.033021409Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.033302505Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.033324325Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-ms892/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.033366926Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-ms892/console-operator-584dcb6b54-ms892.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.033469588Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-ms892/console-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.033482398Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-ms892/console-operator/console-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.033489759Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-ms892/console-operator/console-operator/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.03352493Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-ms892/console-operator/console-operator/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.034050011Z namespaces/openshift-console-operator/pods/console-operator-584dcb6b54-ms892/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.034099122Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.034145203Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.034200394Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.034242685Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.034298826Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.034306006Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.034312206Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.034360028Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.034510411Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.034690135Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.034826047Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.035056602Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.035273937Z namespaces/openshift-console/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.035319278Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.035527422Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.035568393Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.035638225Z namespaces/openshift-console/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.035677156Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.035762327Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.035952431Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.036246648Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.036289459Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.03634596Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.036395091Z namespaces/openshift-console/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.036441642Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.036522314Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.036575375Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.036607276Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.036698698Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.036753459Z namespaces/openshift-console/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.03679993Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.037519195Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.037611777Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.037926414Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.038026966Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.03819279Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.038323622Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.038497026Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.038557557Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.038603568Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.03866758Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.038718331Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.038777712Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.038824093Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.038931205Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.039030717Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.039085159Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.03913244Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.039182271Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.039221712Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.039279053Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.039327814Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.039375735Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.039385815Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.039438876Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.03959322Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.039774663Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.039937727Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.040137741Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.040331495Z namespaces/openshift-console/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.040344265Z namespaces/openshift-console/pods/console-866cdddcb6-mbxzp/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.040394847Z namespaces/openshift-console/pods/console-866cdddcb6-mbxzp/console-866cdddcb6-mbxzp.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.040488339Z namespaces/openshift-console/pods/console-866cdddcb6-mbxzp/console/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.040499849Z namespaces/openshift-console/pods/console-866cdddcb6-mbxzp/console/console/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.040504199Z namespaces/openshift-console/pods/console-866cdddcb6-mbxzp/console/console/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.04054569Z namespaces/openshift-console/pods/console-866cdddcb6-mbxzp/console/console/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.040636132Z namespaces/openshift-console/pods/console-866cdddcb6-mbxzp/console/console/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.040704363Z namespaces/openshift-console/pods/console-866cdddcb6-mbxzp/console/console/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.040733284Z namespaces/openshift-console/pods/downloads-7c97dc4555-98g8r/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.040778575Z namespaces/openshift-console/pods/downloads-7c97dc4555-98g8r/downloads-7c97dc4555-98g8r.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.040868597Z namespaces/openshift-console/pods/downloads-7c97dc4555-98g8r/download-server/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.040875757Z namespaces/openshift-console/pods/downloads-7c97dc4555-98g8r/download-server/download-server/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.040894117Z namespaces/openshift-console/pods/downloads-7c97dc4555-98g8r/download-server/download-server/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.0410003Z namespaces/openshift-console/pods/downloads-7c97dc4555-98g8r/download-server/download-server/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.041405198Z namespaces/openshift-console/pods/downloads-7c97dc4555-98g8r/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.0414718Z namespaces/openshift-console/pods/downloads-7c97dc4555-98g8r/download-server/download-server/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.041508891Z namespaces/openshift-console/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.041542521Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.041601673Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.041642053Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.041786816Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.041833718Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.041901289Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.041966491Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.042025352Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.042073743Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.042156915Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.042238746Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.042325578Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.042493732Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.042536523Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.042601734Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.042644015Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.042730387Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.042776718Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.042830209Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.042936581Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.043010383Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.043058414Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.043180876Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.043262698Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.04334455Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.043424912Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.043504013Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.043626536Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.043784499Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.04383224Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.043869661Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.043968363Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.044014874Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.044086376Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.044125987Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.044212528Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.0442939Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.044330631Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.044381992Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.044428003Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.044464684Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.044515175Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.044522455Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.044563096Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.044706359Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.044914994Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.045070417Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.045278951Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.045476846Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.045496346Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.045558787Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.045598998Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.04570432Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.045747772Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.045792302Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.045831543Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.045903265Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.045947276Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.046043308Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.046120769Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.046200081Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.046280033Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.046316764Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.046392895Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.046427976Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.046513078Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.046544378Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.04659637Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.046676291Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.046736663Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.046756153Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.046803514Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.046908366Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.046985028Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.047196032Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.047275494Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.047357406Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.047438398Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.047519079Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.047642182Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.047797355Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.04801729Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.048063891Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.048114472Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.048160933Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.048211784Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.048228225Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.048324427Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.048392608Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.048435089Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.04848762Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.048522121Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.048562312Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.048603583Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.048612643Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.048659154Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.048802997Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.049116654Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.049255196Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.049484141Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.049663835Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.049709696Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.049749577Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.049792758Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.04989202Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.049948481Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.050031773Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.050067684Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.050142215Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.050177716Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.050270458Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.05034899Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.050431872Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.050568385Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.050598405Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.050658087Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.050688527Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.050779149Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.05083425Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.050865861Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.051002444Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.051067295Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.051112167Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.051212738Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.05129876Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.051381872Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.051463444Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.051550966Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.051673808Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.051857563Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.051964535Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.052016736Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.052061327Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.052105358Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.052145458Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.05218683Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.052269151Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.052346733Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.052400374Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.052429605Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.052487866Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.052516866Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.052592268Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.052632199Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.05267893Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.052723501Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.052754151Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.052760442Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.052799032Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.052983587Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.05317286Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.053309314Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.053523578Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.053802364Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.053840815Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.053930207Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.053975698Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.05407637Z namespaces/openshift-dns/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.054116551Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.054182832Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.054223043Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.054991819Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.055037891Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.055188224Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.055263095Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.055342487Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.055386348Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.055423989Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.05548229Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.055519291Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.055609013Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.055664704Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.055713295Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.055794087Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.055862898Z namespaces/openshift-dns/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.0559367Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.056059773Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.056145624Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.056378329Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.056462731Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.056757927Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.0569052Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.057200727Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.057266858Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.057329009Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.057395191Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.057439872Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.057504043Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.057537264Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.057632836Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.057711558Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.057779789Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.0578333Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.057898452Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.057967953Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.058024944Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.058073476Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.058095186Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.058111826Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.058161707Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.058313591Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.058493415Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.058635808Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.058838162Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.059093187Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.059113358Z namespaces/openshift-dns/pods/dns-default-79nrr/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.059158119Z namespaces/openshift-dns/pods/dns-default-79nrr/dns-default-79nrr.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.059252371Z namespaces/openshift-dns/pods/dns-default-79nrr/dns/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.059262571Z namespaces/openshift-dns/pods/dns-default-79nrr/dns/dns/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.059273371Z namespaces/openshift-dns/pods/dns-default-79nrr/dns/dns/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.059316912Z namespaces/openshift-dns/pods/dns-default-79nrr/dns/dns/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.059410994Z namespaces/openshift-dns/pods/dns-default-79nrr/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.059477376Z namespaces/openshift-dns/pods/dns-default-79nrr/dns/dns/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.059499796Z namespaces/openshift-dns/pods/dns-default-79nrr/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.059513487Z namespaces/openshift-dns/pods/dns-default-79nrr/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.059522607Z namespaces/openshift-dns/pods/dns-default-79nrr/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.059625469Z namespaces/openshift-dns/pods/dns-default-79nrr/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.05968254Z namespaces/openshift-dns/pods/dns-default-79nrr/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.059749032Z namespaces/openshift-dns/pods/dns-default-79nrr/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.059769472Z namespaces/openshift-dns/pods/dns-default-jpzzq/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.059804303Z namespaces/openshift-dns/pods/dns-default-jpzzq/dns-default-jpzzq.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.059903325Z namespaces/openshift-dns/pods/dns-default-jpzzq/dns/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.059926695Z namespaces/openshift-dns/pods/dns-default-jpzzq/dns/dns/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.059934356Z namespaces/openshift-dns/pods/dns-default-jpzzq/dns/dns/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060010087Z namespaces/openshift-dns/pods/dns-default-jpzzq/dns/dns/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060103689Z namespaces/openshift-dns/pods/dns-default-jpzzq/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060177371Z namespaces/openshift-dns/pods/dns-default-jpzzq/dns/dns/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060232142Z namespaces/openshift-dns/pods/dns-default-jpzzq/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060246492Z namespaces/openshift-dns/pods/dns-default-jpzzq/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060252302Z namespaces/openshift-dns/pods/dns-default-jpzzq/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060291993Z namespaces/openshift-dns/pods/dns-default-jpzzq/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060384465Z namespaces/openshift-dns/pods/dns-default-jpzzq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060453136Z namespaces/openshift-dns/pods/dns-default-jpzzq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060510378Z namespaces/openshift-dns/pods/dns-default-t5gxc/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060551849Z namespaces/openshift-dns/pods/dns-default-t5gxc/dns-default-t5gxc.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060636451Z namespaces/openshift-dns/pods/dns-default-t5gxc/dns/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060644681Z namespaces/openshift-dns/pods/dns-default-t5gxc/dns/dns/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060655361Z namespaces/openshift-dns/pods/dns-default-t5gxc/dns/dns/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060716132Z namespaces/openshift-dns/pods/dns-default-t5gxc/dns/dns/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060806264Z namespaces/openshift-dns/pods/dns-default-t5gxc/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060862505Z namespaces/openshift-dns/pods/dns-default-t5gxc/dns/dns/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060939527Z namespaces/openshift-dns/pods/dns-default-t5gxc/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060953417Z namespaces/openshift-dns/pods/dns-default-t5gxc/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.060968198Z namespaces/openshift-dns/pods/dns-default-t5gxc/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.061015879Z namespaces/openshift-dns/pods/dns-default-t5gxc/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.061108701Z namespaces/openshift-dns/pods/dns-default-t5gxc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.061178522Z namespaces/openshift-dns/pods/dns-default-t5gxc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.061217603Z namespaces/openshift-dns/pods/node-resolver-74w4r/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.061254084Z namespaces/openshift-dns/pods/node-resolver-74w4r/node-resolver-74w4r.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.061360096Z namespaces/openshift-dns/pods/node-resolver-74w4r/dns-node-resolver/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.061370996Z namespaces/openshift-dns/pods/node-resolver-74w4r/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.061377756Z namespaces/openshift-dns/pods/node-resolver-74w4r/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.061420217Z namespaces/openshift-dns/pods/node-resolver-74w4r/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.061509129Z namespaces/openshift-dns/pods/node-resolver-74w4r/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.061576291Z namespaces/openshift-dns/pods/node-resolver-74w4r/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.061627752Z namespaces/openshift-dns/pods/node-resolver-pjclb/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.061668233Z namespaces/openshift-dns/pods/node-resolver-pjclb/node-resolver-pjclb.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.061760584Z namespaces/openshift-dns/pods/node-resolver-pjclb/dns-node-resolver/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.061769755Z namespaces/openshift-dns/pods/node-resolver-pjclb/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.061774675Z namespaces/openshift-dns/pods/node-resolver-pjclb/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.061800885Z namespaces/openshift-dns/pods/node-resolver-pjclb/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.061901758Z namespaces/openshift-dns/pods/node-resolver-pjclb/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.06199964Z namespaces/openshift-dns/pods/node-resolver-pjclb/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.06203093Z namespaces/openshift-dns/pods/node-resolver-wlzmw/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.062062971Z namespaces/openshift-dns/pods/node-resolver-wlzmw/node-resolver-wlzmw.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.062147663Z namespaces/openshift-dns/pods/node-resolver-wlzmw/dns-node-resolver/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.062154913Z namespaces/openshift-dns/pods/node-resolver-wlzmw/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.062158913Z namespaces/openshift-dns/pods/node-resolver-wlzmw/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.062191144Z namespaces/openshift-dns/pods/node-resolver-wlzmw/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.062282426Z namespaces/openshift-dns/pods/node-resolver-wlzmw/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.062351687Z namespaces/openshift-dns/pods/node-resolver-wlzmw/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.062388958Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.062422239Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.06247174Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.062508271Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.062604813Z namespaces/openshift-etcd/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.062637053Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.062718155Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.062747596Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.062822807Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.062853138Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.062978441Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.063061452Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.063143834Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.063185695Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.063237586Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.063272787Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.063320008Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.06339879Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.063455451Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.063494212Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.063582554Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.063652795Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.063690176Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.063799428Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.06387543Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.063993183Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.064075194Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.064153836Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.064271729Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.064418441Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.064489003Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.064537044Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.064585065Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.064630266Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.064672607Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.064710328Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.06478974Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.064868631Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.065070346Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.065107686Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.065168078Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.065207789Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.065243839Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.06525723Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.065330031Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.065487724Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.065674848Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.065811211Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.066069747Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.066305932Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.066358213Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.066423584Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.066486706Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.066600258Z namespaces/openshift-host-network/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.066655169Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.066743221Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.066791122Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.066870304Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.066939286Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.067050038Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.06715143Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.067249952Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.067313634Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.067378085Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.067448946Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.067504828Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.06761028Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.067677241Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.067740073Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.067841925Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.067955527Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.068010098Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.068124291Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.068228843Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.068328555Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.068429167Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.06852978Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.068694403Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.068814696Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.068873897Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.068956879Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.069043021Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.069080772Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.069162693Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.069205944Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.069293486Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.069352767Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.069455929Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.069559212Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.069612983Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.069673344Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.069732995Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.069792897Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.069850638Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.069873109Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.06995569Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.070125784Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.070316638Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.070466921Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.070681556Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.070898631Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.070958312Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.071026183Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.071074884Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.071175836Z namespaces/openshift-image-registry/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.071229318Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.071314639Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.07135796Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.071420312Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.071475003Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.071613926Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.071760209Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.072002154Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.072053545Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.072113566Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.072174018Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.072233969Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.072362752Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.072437113Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.072486754Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.072594667Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.072653038Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.072665698Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.072711289Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.072816932Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.072866903Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.073983497Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.074085919Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.074446147Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.074526568Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.074732863Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.074852245Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.07507273Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.075135931Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.075184142Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.075238353Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.075274204Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.075323235Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.075373866Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.075448128Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.075521979Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.075592711Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.075637962Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.075704174Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.075744684Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.075867717Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.075984029Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.076044801Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.076092212Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.076140363Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.076180764Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.076224915Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.076251945Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.076295866Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.07646569Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.076650414Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.076795417Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.077032292Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.077225056Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.077233926Z namespaces/openshift-image-registry/pods/image-registry-659f8869dc-t2rzb/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.077266847Z namespaces/openshift-image-registry/pods/image-registry-659f8869dc-t2rzb/image-registry-659f8869dc-t2rzb.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.077380779Z namespaces/openshift-image-registry/pods/image-registry-659f8869dc-t2rzb/registry/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.07738845Z namespaces/openshift-image-registry/pods/image-registry-659f8869dc-t2rzb/registry/registry/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.07739897Z namespaces/openshift-image-registry/pods/image-registry-659f8869dc-t2rzb/registry/registry/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.077450831Z namespaces/openshift-image-registry/pods/image-registry-659f8869dc-t2rzb/registry/registry/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.079690999Z namespaces/openshift-image-registry/pods/image-registry-659f8869dc-t2rzb/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.0797576Z namespaces/openshift-image-registry/pods/image-registry-659f8869dc-t2rzb/registry/registry/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.079775601Z namespaces/openshift-image-registry/pods/node-ca-2d5z9/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.079829482Z namespaces/openshift-image-registry/pods/node-ca-2d5z9/node-ca-2d5z9.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.079909304Z namespaces/openshift-image-registry/pods/node-ca-2d5z9/node-ca/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.079932214Z namespaces/openshift-image-registry/pods/node-ca-2d5z9/node-ca/node-ca/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.079940934Z namespaces/openshift-image-registry/pods/node-ca-2d5z9/node-ca/node-ca/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.079980775Z namespaces/openshift-image-registry/pods/node-ca-2d5z9/node-ca/node-ca/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.080149479Z namespaces/openshift-image-registry/pods/node-ca-2d5z9/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.08021527Z namespaces/openshift-image-registry/pods/node-ca-2d5z9/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.080246271Z namespaces/openshift-image-registry/pods/node-ca-2thtx/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.080286312Z namespaces/openshift-image-registry/pods/node-ca-2thtx/node-ca-2thtx.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.080358403Z namespaces/openshift-image-registry/pods/node-ca-2thtx/node-ca/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.080370174Z namespaces/openshift-image-registry/pods/node-ca-2thtx/node-ca/node-ca/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.080377414Z namespaces/openshift-image-registry/pods/node-ca-2thtx/node-ca/node-ca/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.080429335Z namespaces/openshift-image-registry/pods/node-ca-2thtx/node-ca/node-ca/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.080617279Z namespaces/openshift-image-registry/pods/node-ca-2thtx/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.08068549Z namespaces/openshift-image-registry/pods/node-ca-2thtx/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.080725131Z namespaces/openshift-image-registry/pods/node-ca-st5g5/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.080762022Z namespaces/openshift-image-registry/pods/node-ca-st5g5/node-ca-st5g5.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.080822823Z namespaces/openshift-image-registry/pods/node-ca-st5g5/node-ca/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.080832763Z namespaces/openshift-image-registry/pods/node-ca-st5g5/node-ca/node-ca/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.080842033Z namespaces/openshift-image-registry/pods/node-ca-st5g5/node-ca/node-ca/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.080906145Z namespaces/openshift-image-registry/pods/node-ca-st5g5/node-ca/node-ca/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.081105219Z namespaces/openshift-image-registry/pods/node-ca-st5g5/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.081177021Z namespaces/openshift-image-registry/pods/node-ca-st5g5/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.081208701Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.081252832Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.081306423Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.081360125Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.081414856Z namespaces/openshift-infra/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.081424416Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.081428726Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.081457227Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.08160799Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.081817324Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.081991818Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.082193142Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.082446188Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.082494279Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.08256315Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.082606111Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.082672883Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.082707453Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.082820136Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.082918368Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.08300526Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.083058551Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.083104002Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.083161883Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.083207994Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.083285616Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.083330157Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.083370558Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.083449529Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.083537151Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.083582262Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.083689354Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.083774546Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.08394677Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.084029872Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.084257167Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.084377519Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.084530332Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.084585324Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.084628045Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.084668986Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.084717427Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.084783898Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.084823709Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.084931571Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.085013513Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.085062344Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.085102665Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.085145916Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.085196027Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.085235728Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.085253168Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.085305729Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.085469493Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.085657517Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.08579014Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.086003194Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.08672502Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.08674635Z namespaces/openshift-ingress-canary/pods/ingress-canary-j7v6r/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.08675678Z namespaces/openshift-ingress-canary/pods/ingress-canary-j7v6r/ingress-canary-j7v6r.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.086845672Z namespaces/openshift-ingress-canary/pods/ingress-canary-j7v6r/serve-healthcheck-canary/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.086859473Z namespaces/openshift-ingress-canary/pods/ingress-canary-j7v6r/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.086866003Z namespaces/openshift-ingress-canary/pods/ingress-canary-j7v6r/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.086955064Z namespaces/openshift-ingress-canary/pods/ingress-canary-j7v6r/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.087055097Z namespaces/openshift-ingress-canary/pods/ingress-canary-j7v6r/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.087139408Z namespaces/openshift-ingress-canary/pods/ingress-canary-j7v6r/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.087175449Z namespaces/openshift-ingress-canary/pods/ingress-canary-ls4sg/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.08721958Z namespaces/openshift-ingress-canary/pods/ingress-canary-ls4sg/ingress-canary-ls4sg.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.087298672Z namespaces/openshift-ingress-canary/pods/ingress-canary-ls4sg/serve-healthcheck-canary/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.087308552Z namespaces/openshift-ingress-canary/pods/ingress-canary-ls4sg/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.087315182Z namespaces/openshift-ingress-canary/pods/ingress-canary-ls4sg/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.087357993Z namespaces/openshift-ingress-canary/pods/ingress-canary-ls4sg/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.087455385Z namespaces/openshift-ingress-canary/pods/ingress-canary-ls4sg/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.087527277Z namespaces/openshift-ingress-canary/pods/ingress-canary-ls4sg/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.087568948Z namespaces/openshift-ingress-canary/pods/ingress-canary-x4n5c/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.087606829Z namespaces/openshift-ingress-canary/pods/ingress-canary-x4n5c/ingress-canary-x4n5c.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.08767119Z namespaces/openshift-ingress-canary/pods/ingress-canary-x4n5c/serve-healthcheck-canary/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.08768931Z namespaces/openshift-ingress-canary/pods/ingress-canary-x4n5c/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.0876964Z namespaces/openshift-ingress-canary/pods/ingress-canary-x4n5c/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.087740832Z namespaces/openshift-ingress-canary/pods/ingress-canary-x4n5c/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.087839814Z namespaces/openshift-ingress-canary/pods/ingress-canary-x4n5c/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.087933976Z namespaces/openshift-ingress-canary/pods/ingress-canary-x4n5c/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.087957126Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.088007187Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.088054828Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.088109979Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.088214102Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.088257992Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.088327074Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.088368145Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.088431446Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.088471517Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.088553509Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.08862853Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.088706522Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.088762613Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.088802604Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.088867725Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.088943597Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.089034809Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.08907973Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.089124841Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.089203503Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.089270984Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.089321755Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.090284336Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.090365758Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.090498711Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.090612043Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.090696805Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.090813807Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.091014492Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.091073743Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.091125254Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.091188205Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.091266157Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.091330348Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.091343039Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.09138773Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.091464911Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.091520512Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.091595854Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.091682246Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.091748127Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.091789178Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.091845729Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.09190349Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.091995772Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.092043773Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.092088005Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.092135615Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.092200977Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.092213777Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.092225917Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.09231645Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.09233044Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.092376851Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.092533124Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.092709878Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.092845641Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.093055005Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.09325039Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.09328562Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.093366492Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.093402863Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.093500525Z namespaces/openshift-ingress/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.093538506Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.093613557Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.093647948Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.093718409Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.09375871Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.093848222Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.094118378Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.094318152Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.094369854Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.094417524Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.094479276Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.094526837Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.094596968Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.09465842Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.094699261Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.094779202Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.094833673Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.094851014Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.094894775Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.094998737Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.095079839Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.095171961Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.095219512Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.095671081Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.095769953Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.096051419Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.096139941Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.096382647Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.096502409Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.096694363Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.096788595Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.096830656Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.096973609Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.096984419Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.09702011Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.097095592Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.097136553Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.097193724Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.097240355Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.097338787Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.097415649Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.09746671Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.097509481Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.097573162Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.097622263Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/openshift-ai-inference-metrics.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.097681765Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.097728286Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.097778176Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.097821468Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.097862498Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.097875439Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.09795089Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.098104634Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.098280227Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.09841589Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.098610344Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.098815309Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.098823229Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.09885665Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/istiod-openshift-gateway-5677ccb49b-4r9gw.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.098988742Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/discovery/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.098999803Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/discovery/discovery/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.099010213Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/discovery/discovery/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.099054524Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/discovery/discovery/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.109760323Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.109830285Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-5677ccb49b-4r9gw/discovery/discovery/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.109902256Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.109929237Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.1100733Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/istio-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.1100834Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/istio-proxy/istio-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.11008861Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.110125381Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.111538491Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.111632463Z namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.111677344Z namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router-default-68787665fc-c2vr7.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.111806197Z namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.111812327Z namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router/router/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.111818637Z namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router/router/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.111872218Z namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router/router/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.111997261Z namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router/router/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.112063663Z namespaces/openshift-ingress/pods/router-default-68787665fc-c2vr7/router/router/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.112111124Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.112146655Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.112210586Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.112245897Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.112345579Z namespaces/openshift-insights/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.11239333Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.112460001Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.112504362Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.112570853Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.112618795Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.112733877Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.112842029Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.113063564Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.113097895Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.113143776Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.113229528Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.113274239Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.11334489Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.113392971Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.113472533Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.113521904Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.113560795Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.113647347Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.113717748Z namespaces/openshift-insights/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.113752079Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.114633038Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.11472229Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.115019426Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.115140319Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.115437025Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.115520987Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.11568253Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.115743541Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.115790282Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.115851354Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.115897175Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.116007957Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.116063058Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.116106679Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.116169341Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.116216992Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.116270733Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.116323104Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.116373605Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.116412506Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.116495608Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.116574099Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.11662992Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.116671981Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.116736513Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.116783364Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.116850595Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.116898906Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.116982558Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.117026219Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.11706832Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.11707966Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.117127501Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.117336946Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.117596231Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.117739014Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.117965819Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.118207444Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.118215834Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-6k54l/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.118254695Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-6k54l/insights-operator-5885fdb759-6k54l.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.118350817Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-6k54l/insights-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.118358388Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-6k54l/insights-operator/insights-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.118371318Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-6k54l/insights-operator/insights-operator/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.118416549Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-6k54l/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.118806037Z namespaces/openshift-insights/pods/insights-operator-5885fdb759-6k54l/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.119140434Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.119171805Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/insights-runtime-extractor-dzw8g.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.119269317Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/exporter/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.119276677Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/exporter/exporter/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.119281067Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/exporter/exporter/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.119314408Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/exporter/exporter/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.11940502Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.119473432Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/exporter/exporter/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.119512232Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/extractor/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.119520142Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/extractor/extractor/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.119524442Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/extractor/extractor/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.119555323Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/extractor/extractor/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.119658665Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.119731157Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/extractor/extractor/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.119753258Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.119759758Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.119768498Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.119822489Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.119934211Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120000733Z namespaces/openshift-insights/pods/insights-runtime-extractor-dzw8g/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120053874Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120088905Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/insights-runtime-extractor-h5pr8.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120174696Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/exporter/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120182717Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/exporter/exporter/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120190747Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/exporter/exporter/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120237328Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/exporter/exporter/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.12032351Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120396051Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/exporter/exporter/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120430912Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/extractor/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120441222Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/extractor/extractor/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120445212Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/extractor/extractor/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120490503Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/extractor/extractor/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120590115Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120659557Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/extractor/extractor/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120693667Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120703178Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120707278Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120747969Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120835731Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120945893Z namespaces/openshift-insights/pods/insights-runtime-extractor-h5pr8/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.120985154Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121024305Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/insights-runtime-extractor-mxrqp.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121106167Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/exporter/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121113416Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/exporter/exporter/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121119487Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/exporter/exporter/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121165628Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/exporter/exporter/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.12126105Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121338681Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/exporter/exporter/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121371082Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/extractor/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121377772Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/extractor/extractor/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121388262Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/extractor/extractor/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121429573Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/extractor/extractor/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121526396Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121588767Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/extractor/extractor/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121620398Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121630718Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121640198Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121669188Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.12175749Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121831372Z namespaces/openshift-insights/pods/insights-runtime-extractor-mxrqp/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121854123Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121925894Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.121975435Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.122016096Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.122064867Z namespaces/openshift-keda/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.122071777Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.122080677Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.122128408Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.12221407Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.122259501Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.122268261Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.122295662Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.122354853Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.122396334Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.122491926Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.122570458Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.122625129Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.122632249Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.12267364Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.122813093Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.123082279Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.123273493Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.123463887Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.12360467Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.123778724Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.123813974Z namespaces/openshift-keda/operators.coreos.com/installplans/install-r24ld.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.123948327Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.123989908Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.124041819Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.12408Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.124137792Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.124176422Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.124271274Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.124315215Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.124386147Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.124427258Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.124471689Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.124507069Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.124592271Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.124666543Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.124747354Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.124798006Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.124832926Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.124902198Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.124946109Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.125028851Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.125071732Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.125112472Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.125192774Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.125249445Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.125294216Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.125419189Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.12549399Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.125572962Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.125653104Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.125727195Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.125850998Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.126069183Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.126118204Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.126158665Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.126201816Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.126250287Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.126293308Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.126336559Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.12641697Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.126495922Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.126545203Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.126577804Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.126633785Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.126672436Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.126725427Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.126733037Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.126765478Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.126941801Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.127163416Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.12731926Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.127520874Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.127702118Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.127731058Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.127785219Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.12782577Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.127934143Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.127980784Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.128037215Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.128087666Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.128150517Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.128198338Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.128296181Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.128376592Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.128454714Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.128541386Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.128571796Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.128629278Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.128670558Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.128830322Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.128869503Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.128938274Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.129015466Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.129077967Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.129124118Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.12921568Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.129292602Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.129391994Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.129470896Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.129563858Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.12968651Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.129839944Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.129891875Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.129965596Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.130025378Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.130061059Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.13011999Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.13015635Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.130241562Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.130315664Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.130364095Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.130409766Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.130469267Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.130503978Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.13060061Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.130653661Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.130701512Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.130756083Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.130787394Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.130849345Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.130859716Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.130919757Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.131106861Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.131302005Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.131439748Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.131626992Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.131818416Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.131854087Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.131935368Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.131968779Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.132034771Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.132074911Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.132140783Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.132178124Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.132234545Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.132273026Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.132354697Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.132430799Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.132512101Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.132547102Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.132594493Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.132642984Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.132685164Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.132763426Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.132814487Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.132852118Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.13295167Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.133008142Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.133052343Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.133158005Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.133239507Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.133332669Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.13341165Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.133493472Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.133613364Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.133785688Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.13384618Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.133895401Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.134041814Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.134087985Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.134141456Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.134178507Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.134267318Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.1343419Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.134391521Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.134433222Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.134490293Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.134525694Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.134596246Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.134633506Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.134682827Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.134691688Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.134735259Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.134898292Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.135098066Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.135236349Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.135434834Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.135616527Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.135651868Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.135711969Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.13574455Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.135813512Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.135853792Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.135942184Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.135968265Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.136037887Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.136076577Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.136161319Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.136238671Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.136319413Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.136369443Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.136410055Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.136461565Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.136514797Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.136586848Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.136638469Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.13667679Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.136758972Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.136798833Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.136805973Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.136848294Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.136932396Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.136976827Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.137079669Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.137157071Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.137665701Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.137740443Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.137826715Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.137967678Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.138121541Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.138171602Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.138216713Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.138261474Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.138308615Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.138375897Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.138422027Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.138505019Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.138587101Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.138624442Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.138669293Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.138722634Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.138755605Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.138804516Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.138813676Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.138848497Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.139026391Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.139205844Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.139344317Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.139543831Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.139742976Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.139791677Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.139829798Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.139869629Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.139961991Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.140005191Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.140062903Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.140108094Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.140169345Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.140192036Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.140270427Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.140358289Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.140441701Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.140475691Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.140524832Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.140567933Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.140611434Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.140692776Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.140734857Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.140779608Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.14086087Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.140943861Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.140990593Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.141100625Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.141177547Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.141261178Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.14135277Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.141432882Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.141557545Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.141722038Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.141772139Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.14181349Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.141862621Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.141933673Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.141988994Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.142037255Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.142121357Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.142198288Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.14224118Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.14228606Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.142347502Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.142383952Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.142417923Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.142428123Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.142474814Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.142624808Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.142803492Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.143038306Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.14322206Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.143405704Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.143448215Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.143505916Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.143544837Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.143611529Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.143646159Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.143711231Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.143749582Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.143803803Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.143841364Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.143947436Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.144029638Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.14411864Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.144164911Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.144206042Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.144259343Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.144303673Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.144381265Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.144425336Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.144470237Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.144552979Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.14460594Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.144646881Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.144741753Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.144823075Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.144932347Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.145016499Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.145100871Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.145224133Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.145398897Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.145452238Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.145500219Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.14554265Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.145577631Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.145637272Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.145674833Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.145755015Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.145834926Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.145872647Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.145951399Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.14599443Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.146040701Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.146075462Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.146087772Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.146139083Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.146287556Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.14646442Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.146602383Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.146800497Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.147020432Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.147059813Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.147121214Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.147156385Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.147242987Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.147283737Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.147356579Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.14738927Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.147451511Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.147490462Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.147576504Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.147693036Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.147795518Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.147846889Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.147896161Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.147978552Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.148028163Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.148119705Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.148174416Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.148221117Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.148299429Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.14834579Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.14835878Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.148407712Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.148486953Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.148535904Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.148647737Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.148730668Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.148930803Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.149017025Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.149138937Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.14926305Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.149498045Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.149561186Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.149610217Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.149661318Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.149703229Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.149758921Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.149813472Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.149901373Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.150023986Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.150078307Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.150128168Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.15018123Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.150235901Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.150283842Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.150296082Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.150341043Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.150489966Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.15066916Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.150806183Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.151060989Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.151262503Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.151270153Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.151313364Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.151456257Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2/kube-storage-version-migrator-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.151464817Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.151477227Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.151521908Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.151634601Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.151764704Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.151797004Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.151844545Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.151889886Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.151974698Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.152026119Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.152085311Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.152124241Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.152173842Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.152208063Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.152297225Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.152446258Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.15254228Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.152592861Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.152636772Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.152686373Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.152716064Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.152806176Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.152862967Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.152937429Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.153040831Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.153088592Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.153127283Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.153231565Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.153327107Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.15345554Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.153527991Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.153645374Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.153768176Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.153976481Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.154035032Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.154083473Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.154135524Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.154170835Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.154226496Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.154278267Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.154357959Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.154443291Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.154490242Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.154532503Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.154577874Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.154620735Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.154668846Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.154675716Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.154720537Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.15486969Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.155099745Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.155262688Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.155523774Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.155724188Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.155731798Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-rkmdz/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.15577581Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-rkmdz/migrator-6865c9967-rkmdz.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.155854251Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-rkmdz/graceful-termination/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.155862001Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-rkmdz/graceful-termination/graceful-termination/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.155869941Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-rkmdz/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.155949993Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-rkmdz/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.156054116Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-rkmdz/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.156124747Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-rkmdz/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.156162388Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-rkmdz/migrator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.156172328Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-rkmdz/migrator/migrator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.156179798Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-rkmdz/migrator/migrator/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.156207319Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-rkmdz/migrator/migrator/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.156309041Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-rkmdz/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.156375542Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-6865c9967-rkmdz/migrator/migrator/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.156416143Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.156481705Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.156535926Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.156577437Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.156638568Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.156691569Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.156761411Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.156794891Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.156849453Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.156891743Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.157018706Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.157161539Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.157283462Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.157353273Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.157404094Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.157454155Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.157486926Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.157572268Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.157618639Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.15766253Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.157745902Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.157796593Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.157805203Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.157837474Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.158029908Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.158087589Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.15813014Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.158242392Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.158335814Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.158640361Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.158716653Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.158906967Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.15907694Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.159276694Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.159342346Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.159387387Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.159454858Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.15951891Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.159581451Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.159627282Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.159715744Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.159787895Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.159841106Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.159900448Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.1599864Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.160034081Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.160108042Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.160151633Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.160208305Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.160217315Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.160265136Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.160425749Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.160600353Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.160744506Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.161051752Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.161236767Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.161502362Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.161545913Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-fx6q4.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.161628115Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.161668186Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.161719557Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.161757138Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.161813859Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.161842349Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.161945612Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.161956202Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78db6c7dfd-gspjc/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.162007503Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78db6c7dfd-gspjc/lws-controller-manager-78db6c7dfd-gspjc.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.162093385Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78db6c7dfd-gspjc/manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.162106385Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78db6c7dfd-gspjc/manager/manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.162112775Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78db6c7dfd-gspjc/manager/manager/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.162157596Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78db6c7dfd-gspjc/manager/manager/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.162279039Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78db6c7dfd-gspjc/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.16234882Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78db6c7dfd-gspjc/manager/manager/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.162394921Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78db6c7dfd-p54hf/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.162422602Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78db6c7dfd-p54hf/lws-controller-manager-78db6c7dfd-p54hf.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.162505564Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78db6c7dfd-p54hf/manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.162513364Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78db6c7dfd-p54hf/manager/manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.162517414Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78db6c7dfd-p54hf/manager/manager/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.162556425Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78db6c7dfd-p54hf/manager/manager/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.162904892Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78db6c7dfd-p54hf/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.163034655Z namespaces/openshift-lws-operator/pods/lws-controller-manager-78db6c7dfd-p54hf/manager/manager/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.163065026Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-m8rzf/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.163120137Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-m8rzf/openshift-lws-operator-fd8ccff4c-m8rzf.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.163201599Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-m8rzf/openshift-lws-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.163209919Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-m8rzf/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.163216149Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-m8rzf/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.16326395Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-m8rzf/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.163424073Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-m8rzf/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.163496225Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-m8rzf/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.163525556Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.163563846Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.163615067Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.163648478Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.163700089Z namespaces/openshift-machine-api/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.16371184Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.16371586Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.16373675Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.163904204Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.164112318Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.164250161Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.164445755Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.1646411Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.16465157Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.16465745Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.164687201Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.164788283Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.164860534Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.164869314Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.164942676Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.16512279Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.165303734Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.165458057Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.165656041Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.165845175Z namespaces/openshift-marketplace/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.165855255Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.165860276Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.165891816Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.16606255Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.16607019Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.166106221Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.166197103Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.166279384Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.166370317Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.166418807Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.166457258Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.166621222Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.166808226Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.166974109Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.167177134Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.167387228Z namespaces/openshift-monitoring/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.167422129Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.167492621Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.167541222Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.167578592Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.167628533Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.167762876Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.168222626Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.168641925Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.16886884Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.168932421Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.168990523Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.169027383Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.169116485Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.169174437Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.169220698Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.169296759Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.169354461Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.169397391Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.172500588Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.172647681Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.173653293Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.173732834Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.174732246Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.174863659Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.17540588Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.175568334Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.175611215Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.175755328Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.175782198Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.175837689Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.175894271Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.175986533Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.176080925Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.176124466Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.176168447Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.176296159Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.17633651Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.176412522Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.176449253Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.176559065Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.176606136Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.176708928Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.176903612Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.177016525Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.17726367Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.177415403Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.177535996Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.177625198Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.17773125Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.177818452Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.177893873Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.177966475Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.178079497Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.178148049Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.178233561Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.178335873Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.178417135Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.178594519Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.178687661Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.178782452Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.178870194Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.178997577Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.179086879Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.17915572Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.179201201Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.179253703Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.179260983Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.179302824Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.179495388Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.179731313Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.179877636Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.180121911Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.180329296Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.180377927Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.180461599Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.180473769Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18053981Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.180714434Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.180723484Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.180728314Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.180761315Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.180859947Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.180959199Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18100518Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18101344Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.181018171Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.181062881Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.181155073Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.181225785Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.181260916Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.181279706Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.181289006Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.181313057Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.181415429Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18148973Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.181530291Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.181540852Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.181546782Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.181580392Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.181672665Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.181738556Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.181783667Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.181794137Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.181800387Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.181829058Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.182023062Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.182093643Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.182137824Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.182147645Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.182153575Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.182188925Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.182277127Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.182348099Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18238328Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18239091Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18239491Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.182438251Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.182521773Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.182588084Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.182626305Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-9vgj6/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.182671566Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-9vgj6/cluster-monitoring-operator-548b489cb8-9vgj6.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.182762238Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-9vgj6/cluster-monitoring-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.182772918Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-9vgj6/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.182779318Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-9vgj6/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.182814739Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-9vgj6/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.183635767Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-9vgj6/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.183708948Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-548b489cb8-9vgj6/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.183755819Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18379886Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/kube-state-metrics-56d9f944b4-xr9dr.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.183919613Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/kube-rbac-proxy-main/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.183932853Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.183941523Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.183994274Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.184116317Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.184193019Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.184227489Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/kube-rbac-proxy-self/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.184242009Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.1842479Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18428085Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.184369662Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.184441764Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.184471024Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/kube-state-metrics/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.184478974Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.184486535Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.184516565Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18472443Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.184792571Z namespaces/openshift-monitoring/pods/kube-state-metrics-56d9f944b4-xr9dr/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.184834272Z namespaces/openshift-monitoring/pods/metrics-server-c5b67497c-gr5jc/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.184863093Z namespaces/openshift-monitoring/pods/metrics-server-c5b67497c-gr5jc/metrics-server-c5b67497c-gr5jc.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.185005846Z namespaces/openshift-monitoring/pods/metrics-server-c5b67497c-gr5jc/metrics-server/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.185016526Z namespaces/openshift-monitoring/pods/metrics-server-c5b67497c-gr5jc/metrics-server/metrics-server/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.185020576Z namespaces/openshift-monitoring/pods/metrics-server-c5b67497c-gr5jc/metrics-server/metrics-server/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.185058467Z namespaces/openshift-monitoring/pods/metrics-server-c5b67497c-gr5jc/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.185151339Z namespaces/openshift-monitoring/pods/metrics-server-c5b67497c-gr5jc/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.185222041Z namespaces/openshift-monitoring/pods/metrics-server-c5b67497c-gr5jc/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.185257071Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-5bx4k/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.185289192Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-5bx4k/monitoring-plugin-68b7fc9574-5bx4k.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.185394684Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-5bx4k/monitoring-plugin/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.185403025Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-5bx4k/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.185406914Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-5bx4k/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.185447505Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-5bx4k/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.185545197Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-5bx4k/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.185607209Z namespaces/openshift-monitoring/pods/monitoring-plugin-68b7fc9574-5bx4k/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18565501Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18568978Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/node-exporter-jlpsc.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.185806913Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/init-textfile/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.185817033Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/init-textfile/init-textfile/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.185822723Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/init-textfile/init-textfile/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.185858974Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.185966717Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.186049508Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.186090579Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.186104349Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18611061Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.186159081Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.186256853Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.186328934Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.186375385Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/node-exporter/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.186385026Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/node-exporter/node-exporter/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.186396356Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/node-exporter/node-exporter/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.186434316Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.186557199Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.186620771Z namespaces/openshift-monitoring/pods/node-exporter-jlpsc/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.186680362Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.186723813Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/node-exporter-ks6hc.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.186827975Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/init-textfile/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.186836025Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/init-textfile/init-textfile/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.186840095Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/init-textfile/init-textfile/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.186890856Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.186999479Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18706879Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.187109581Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.187118211Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.187122291Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.187148222Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.187238184Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.187305195Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.187342456Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/node-exporter/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.187351866Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/node-exporter/node-exporter/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.187357866Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/node-exporter/node-exporter/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.187397827Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18750703Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.187573091Z namespaces/openshift-monitoring/pods/node-exporter-ks6hc/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.187617162Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.187657003Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/node-exporter-tfknf.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.187750705Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/init-textfile/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.187758135Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/init-textfile/init-textfile/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.187762645Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/init-textfile/init-textfile/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.187803376Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.187892668Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.187977049Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18800448Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18801352Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18801757Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.188063611Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.188162133Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.188228315Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.188267096Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/node-exporter/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.188275376Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/node-exporter/node-exporter/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.188279366Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/node-exporter/node-exporter/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.188313677Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.188418699Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.188486571Z namespaces/openshift-monitoring/pods/node-exporter-tfknf/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.188530131Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.188566232Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/openshift-state-metrics-b8f49f5c6-47wrj.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.188702665Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/kube-rbac-proxy-main/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.188713315Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.188719116Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.188756816Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.188850938Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18895224Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.188992111Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/kube-rbac-proxy-self/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.188999871Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.189006282Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.189058743Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.189154505Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.189220636Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.189259337Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/openshift-state-metrics/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.189276167Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.189290058Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.189339519Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.189449541Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.189519383Z namespaces/openshift-monitoring/pods/openshift-state-metrics-b8f49f5c6-47wrj/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.189581174Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.189646935Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.189835139Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.189842389Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.18984837Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.189892451Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190061904Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190131706Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190157146Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190163567Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190169316Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190220978Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.19031506Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190384671Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190417612Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190424332Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190428282Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190467353Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190558075Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190625416Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190651037Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190658157Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190671297Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190706158Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.19079387Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190861931Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190920543Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190934673Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190940753Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.190994544Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.191088756Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.191158418Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.191201539Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.191211689Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.191217669Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.19124963Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.191969265Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.192049077Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.192085748Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.192094478Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.192098298Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.192140849Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.192241891Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.192312313Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.192354753Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-klb4j/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.192399954Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-klb4j/prometheus-operator-84d69f794d-klb4j.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.192488016Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-klb4j/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.192495036Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-klb4j/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.192498706Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-klb4j/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.192537377Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-klb4j/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.192624479Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-klb4j/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.192691451Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-klb4j/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.192721471Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-klb4j/prometheus-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.192731122Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-klb4j/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.192740632Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-klb4j/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.192780783Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-klb4j/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.193067819Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-klb4j/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.1931349Z namespaces/openshift-monitoring/pods/prometheus-operator-84d69f794d-klb4j/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.193174951Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-wxf69/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.193216432Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-wxf69/prometheus-operator-admission-webhook-7f8d78f54b-wxf69.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.193324534Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-wxf69/prometheus-operator-admission-webhook/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.193342535Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-wxf69/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.193349485Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-wxf69/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.193391625Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-wxf69/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.193481858Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-wxf69/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.193550289Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7f8d78f54b-wxf69/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.19358683Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.193636071Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/telemeter-client-58476696fd-pthfz.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.193772514Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.193785954Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.193789874Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.193828165Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.193940747Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.194023899Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.194098601Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/reload/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.194103001Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/reload/reload/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.194106861Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/reload/reload/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.194119761Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/reload/reload/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.194216793Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.194287675Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/reload/reload/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.194339416Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/telemeter-client/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.194347256Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/telemeter-client/telemeter-client/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.194351476Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.194389697Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.194479949Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.19454965Z namespaces/openshift-monitoring/pods/telemeter-client-58476696fd-pthfz/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.194587821Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.194650672Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/thanos-querier-5988bb7987-nsstg.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.194778575Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy-metrics/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.194785826Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.194790526Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.194830727Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.194951669Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195032661Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195125743Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy-rules/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195133443Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195140873Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195178544Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195272726Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195341147Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195376278Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy-web/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195383408Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195387628Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195431649Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195517511Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195585723Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195627484Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195634604Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195638534Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195678215Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195765296Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195833318Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195865749Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/prom-label-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.195875769Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.19590097Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.19594336Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.196040682Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.196112944Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.196143215Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/thanos-query/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.196150745Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/thanos-query/thanos-query/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.196159755Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/thanos-query/thanos-query/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.196198666Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.196300248Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.196372349Z namespaces/openshift-monitoring/pods/thanos-querier-5988bb7987-nsstg/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.19641607Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.196449111Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.196494932Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.196541603Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.196662176Z namespaces/openshift-multus/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.196706077Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.196770598Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.196815729Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.19685804Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.196922711Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.197184507Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.197266609Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.197366231Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.197403202Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.197450793Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.197645607Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.197757269Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.19781916Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.197857721Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.197997324Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.198040315Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.198126797Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.198166398Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.198213529Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.198292861Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.198349672Z namespaces/openshift-multus/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.198394573Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.198526496Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.198608937Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.199118618Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.199239541Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.199927076Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.200035228Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.200170851Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.200225372Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.200281763Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.200373455Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.200469197Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.200550709Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.200649041Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.200722353Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.200821065Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.200932317Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.201037299Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.20108553Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.201128301Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.201185372Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.201229284Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.201282295Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.201328986Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.201392367Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.201435668Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.20151429Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.201593901Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.201647553Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.201684753Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.201748235Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.201785925Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.201836037Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.201875987Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.201960599Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.20197837Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.202026691Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.202179674Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.202372238Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.202514011Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.202710085Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.202897639Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.20291732Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.202990361Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/multus-additional-cni-plugins-4cdnz.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.203138225Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/bond-cni-plugin/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.203145555Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.203152945Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.203192616Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.203282728Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.203356029Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.20338609Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/cni-plugins/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.20339126Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/cni-plugins/cni-plugins/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.20339726Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.203444631Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.203537413Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.203611615Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.203644875Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/egress-router-binary-copy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.203651905Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.203655905Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.203694046Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.203794528Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.203893321Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.203949502Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.203960292Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204003363Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204054344Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204130076Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204195577Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204240798Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/routeoverride-cni/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204252498Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204259958Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204268249Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204364941Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204432972Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204476533Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/whereabouts-cni-bincopy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204485423Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204489503Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204507364Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204600706Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204669967Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204702998Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/whereabouts-cni/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204710688Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204714728Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204749159Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204835681Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204930843Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4cdnz/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.204976714Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.205031085Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/multus-additional-cni-plugins-gwwvx.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.205158828Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/bond-cni-plugin/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.205169798Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.205173788Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.205209059Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.205289111Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.205383093Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.205407413Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/cni-plugins/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.205413703Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/cni-plugins/cni-plugins/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.205417393Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.205463834Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.205549786Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.205622258Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.205645558Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/egress-router-binary-copy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.205651348Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.205655428Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.205695219Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.205796051Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.205867033Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206039157Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206049957Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206059747Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206086678Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206159029Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206226491Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206250931Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/routeoverride-cni/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206257311Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206267172Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206304312Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206396424Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206465866Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206508087Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/whereabouts-cni-bincopy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206518277Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206523937Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206548938Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206632979Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206700571Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206740282Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/whereabouts-cni/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206749822Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206756482Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206788483Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206894675Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.206987797Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-gwwvx/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207031968Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207078189Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/multus-additional-cni-plugins-sf8nb.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207206322Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/bond-cni-plugin/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207213572Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207217432Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207262513Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207353685Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207421126Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207488488Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/cni-plugins/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207498788Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/cni-plugins/cni-plugins/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207504748Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207516308Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207618221Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207685492Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207714343Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/egress-router-binary-copy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207724573Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207735853Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207762914Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207852715Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207948608Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.207998069Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208005019Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208009079Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.20804713Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208114351Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208192653Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208215983Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/routeoverride-cni/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208220823Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208225763Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208260744Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208351956Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208423688Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208451868Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/whereabouts-cni-bincopy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208458568Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208462639Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208498029Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208588061Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208661293Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208686163Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/whereabouts-cni/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208692204Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208698144Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208736505Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208821716Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208896448Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-sf8nb/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.208961519Z namespaces/openshift-multus/pods/multus-chg9p/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.20901882Z namespaces/openshift-multus/pods/multus-chg9p/multus-chg9p.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.209100562Z namespaces/openshift-multus/pods/multus-chg9p/kube-multus/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.209107612Z namespaces/openshift-multus/pods/multus-chg9p/kube-multus/kube-multus/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.209111433Z namespaces/openshift-multus/pods/multus-chg9p/kube-multus/kube-multus/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.209146033Z namespaces/openshift-multus/pods/multus-chg9p/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.210061983Z namespaces/openshift-multus/pods/multus-chg9p/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.210136115Z namespaces/openshift-multus/pods/multus-chg9p/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.210163125Z namespaces/openshift-multus/pods/multus-hgt5v/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.210210786Z namespaces/openshift-multus/pods/multus-hgt5v/multus-hgt5v.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.210301198Z namespaces/openshift-multus/pods/multus-hgt5v/kube-multus/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.210308078Z namespaces/openshift-multus/pods/multus-hgt5v/kube-multus/kube-multus/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.210311838Z namespaces/openshift-multus/pods/multus-hgt5v/kube-multus/kube-multus/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.210350999Z namespaces/openshift-multus/pods/multus-hgt5v/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.211175827Z namespaces/openshift-multus/pods/multus-hgt5v/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.211248608Z namespaces/openshift-multus/pods/multus-hgt5v/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.211276899Z namespaces/openshift-multus/pods/multus-qqq2z/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.21131111Z namespaces/openshift-multus/pods/multus-qqq2z/multus-qqq2z.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.211405352Z namespaces/openshift-multus/pods/multus-qqq2z/kube-multus/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.211412422Z namespaces/openshift-multus/pods/multus-qqq2z/kube-multus/kube-multus/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.211416482Z namespaces/openshift-multus/pods/multus-qqq2z/kube-multus/kube-multus/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.211458673Z namespaces/openshift-multus/pods/multus-qqq2z/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.212061146Z namespaces/openshift-multus/pods/multus-qqq2z/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.212134367Z namespaces/openshift-multus/pods/multus-qqq2z/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.212164908Z namespaces/openshift-multus/pods/network-metrics-daemon-9qd9t/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.212209439Z namespaces/openshift-multus/pods/network-metrics-daemon-9qd9t/network-metrics-daemon-9qd9t.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.212315261Z namespaces/openshift-multus/pods/network-metrics-daemon-9qd9t/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.212323871Z namespaces/openshift-multus/pods/network-metrics-daemon-9qd9t/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.212329102Z namespaces/openshift-multus/pods/network-metrics-daemon-9qd9t/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.212357402Z namespaces/openshift-multus/pods/network-metrics-daemon-9qd9t/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.212460604Z namespaces/openshift-multus/pods/network-metrics-daemon-9qd9t/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.212527706Z namespaces/openshift-multus/pods/network-metrics-daemon-9qd9t/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.212563667Z namespaces/openshift-multus/pods/network-metrics-daemon-9qd9t/network-metrics-daemon/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.212571167Z namespaces/openshift-multus/pods/network-metrics-daemon-9qd9t/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.212577057Z namespaces/openshift-multus/pods/network-metrics-daemon-9qd9t/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.212615358Z namespaces/openshift-multus/pods/network-metrics-daemon-9qd9t/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.212789581Z namespaces/openshift-multus/pods/network-metrics-daemon-9qd9t/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.212859323Z namespaces/openshift-multus/pods/network-metrics-daemon-9qd9t/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.212913794Z namespaces/openshift-multus/pods/network-metrics-daemon-dx5sm/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.212972905Z namespaces/openshift-multus/pods/network-metrics-daemon-dx5sm/network-metrics-daemon-dx5sm.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.213056187Z namespaces/openshift-multus/pods/network-metrics-daemon-dx5sm/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.213066977Z namespaces/openshift-multus/pods/network-metrics-daemon-dx5sm/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.213070957Z namespaces/openshift-multus/pods/network-metrics-daemon-dx5sm/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.213100598Z namespaces/openshift-multus/pods/network-metrics-daemon-dx5sm/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.21318969Z namespaces/openshift-multus/pods/network-metrics-daemon-dx5sm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.213258962Z namespaces/openshift-multus/pods/network-metrics-daemon-dx5sm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.213298092Z namespaces/openshift-multus/pods/network-metrics-daemon-dx5sm/network-metrics-daemon/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.213307052Z namespaces/openshift-multus/pods/network-metrics-daemon-dx5sm/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.213313263Z namespaces/openshift-multus/pods/network-metrics-daemon-dx5sm/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.213364764Z namespaces/openshift-multus/pods/network-metrics-daemon-dx5sm/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.213518587Z namespaces/openshift-multus/pods/network-metrics-daemon-dx5sm/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.213585628Z namespaces/openshift-multus/pods/network-metrics-daemon-dx5sm/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.213624369Z namespaces/openshift-multus/pods/network-metrics-daemon-qbb85/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.21366213Z namespaces/openshift-multus/pods/network-metrics-daemon-qbb85/network-metrics-daemon-qbb85.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.213747492Z namespaces/openshift-multus/pods/network-metrics-daemon-qbb85/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.213754712Z namespaces/openshift-multus/pods/network-metrics-daemon-qbb85/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.213758742Z namespaces/openshift-multus/pods/network-metrics-daemon-qbb85/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.213796403Z namespaces/openshift-multus/pods/network-metrics-daemon-qbb85/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.213903355Z namespaces/openshift-multus/pods/network-metrics-daemon-qbb85/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.214071209Z namespaces/openshift-multus/pods/network-metrics-daemon-qbb85/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.214098089Z namespaces/openshift-multus/pods/network-metrics-daemon-qbb85/network-metrics-daemon/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.214105209Z namespaces/openshift-multus/pods/network-metrics-daemon-qbb85/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.21411708Z namespaces/openshift-multus/pods/network-metrics-daemon-qbb85/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.214163561Z namespaces/openshift-multus/pods/network-metrics-daemon-qbb85/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.214325584Z namespaces/openshift-multus/pods/network-metrics-daemon-qbb85/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.214396236Z namespaces/openshift-multus/pods/network-metrics-daemon-qbb85/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.214433736Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.214474768Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.214516888Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.214524118Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.214565179Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.214654111Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.214701552Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.214745803Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.214833445Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.214897086Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.214955708Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.215013579Z namespaces/openshift-must-gather-6q4dh/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.215021549Z namespaces/openshift-must-gather-6q4dh/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.215025579Z namespaces/openshift-must-gather-6q4dh/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.21506946Z namespaces/openshift-must-gather-6q4dh/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.215224773Z namespaces/openshift-must-gather-6q4dh/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.215403997Z namespaces/openshift-must-gather-6q4dh/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.21554039Z namespaces/openshift-must-gather-6q4dh/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.215746055Z namespaces/openshift-must-gather-6q4dh/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.215969149Z namespaces/openshift-network-console/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.21601971Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.216082832Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.216120293Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.216167394Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.216207165Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.216292056Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.216414899Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.216515041Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.216565862Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.216590113Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.216671475Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.216707045Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.216756296Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.216782577Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.216870989Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.21694901Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.217007012Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.217091544Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.217133054Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.217179846Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.217288848Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.21738209Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.217498442Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.217574364Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.217714307Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.21783596Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.217988523Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.218050424Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.218092985Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.218144236Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.218184067Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.218238908Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.218284749Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.21833182Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.218371391Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.218408132Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.218455863Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.218535855Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.218614416Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.218661267Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.218711158Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.218755509Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.21879854Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.218845761Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.218855931Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.218895052Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.219138127Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.219433084Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.219575867Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.219767421Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.219980555Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.219996956Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-q5n5g/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.220028847Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-q5n5g/networking-console-plugin-74c9f884b7-q5n5g.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.220110468Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-q5n5g/networking-console-plugin/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.220118698Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-q5n5g/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.220123189Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-q5n5g/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.220164549Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-q5n5g/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.220255221Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-q5n5g/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.220324943Z namespaces/openshift-network-console/pods/networking-console-plugin-74c9f884b7-q5n5g/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.220361294Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.220398524Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.220442995Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.220487366Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.220536207Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.220572288Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.22063902Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.22067292Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.220720111Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.220760842Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.220870454Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.221007628Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.22111338Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.22114387Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.221205552Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.221255723Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.221291763Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.221370955Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.221400136Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.221453807Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.221499628Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.22158232Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.221622461Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.221666611Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.221742093Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.221800205Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.221832105Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.222040719Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.222131592Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.222321405Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.222397407Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.222609302Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.222694184Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.222794046Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.222855787Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.222908048Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.22297391Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.223026901Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.223114413Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.223165324Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.223210725Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.223270566Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.223298457Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.223349518Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.223389328Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.22347238Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.223549222Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.223590313Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.223631224Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.223686195Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.223723976Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.223778497Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.223818248Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.223862209Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.223874079Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.22393401Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.224089343Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.224293268Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.224437391Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.224639845Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.224949612Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.224964392Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-pzk4p/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225003543Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-pzk4p/network-check-source-7d88cb55d8-pzk4p.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225086925Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-pzk4p/check-endpoints/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225097485Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-pzk4p/check-endpoints/check-endpoints/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225103275Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-pzk4p/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225137686Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-pzk4p/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225272409Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-pzk4p/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225354671Z namespaces/openshift-network-diagnostics/pods/network-check-source-7d88cb55d8-pzk4p/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225387081Z namespaces/openshift-network-diagnostics/pods/network-check-target-d8c8n/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225434122Z namespaces/openshift-network-diagnostics/pods/network-check-target-d8c8n/network-check-target-d8c8n.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225500504Z namespaces/openshift-network-diagnostics/pods/network-check-target-d8c8n/network-check-target-container/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225509324Z namespaces/openshift-network-diagnostics/pods/network-check-target-d8c8n/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225515354Z namespaces/openshift-network-diagnostics/pods/network-check-target-d8c8n/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225547615Z namespaces/openshift-network-diagnostics/pods/network-check-target-d8c8n/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225638417Z namespaces/openshift-network-diagnostics/pods/network-check-target-d8c8n/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225705538Z namespaces/openshift-network-diagnostics/pods/network-check-target-d8c8n/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225742019Z namespaces/openshift-network-diagnostics/pods/network-check-target-fnv6c/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225770479Z namespaces/openshift-network-diagnostics/pods/network-check-target-fnv6c/network-check-target-fnv6c.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225854891Z namespaces/openshift-network-diagnostics/pods/network-check-target-fnv6c/network-check-target-container/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225862292Z namespaces/openshift-network-diagnostics/pods/network-check-target-fnv6c/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.225866252Z namespaces/openshift-network-diagnostics/pods/network-check-target-fnv6c/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.226001095Z namespaces/openshift-network-diagnostics/pods/network-check-target-fnv6c/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.226098037Z namespaces/openshift-network-diagnostics/pods/network-check-target-fnv6c/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.226168498Z namespaces/openshift-network-diagnostics/pods/network-check-target-fnv6c/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.226201089Z namespaces/openshift-network-diagnostics/pods/network-check-target-zzc9f/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.22624558Z namespaces/openshift-network-diagnostics/pods/network-check-target-zzc9f/network-check-target-zzc9f.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.226315851Z namespaces/openshift-network-diagnostics/pods/network-check-target-zzc9f/network-check-target-container/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.226325971Z namespaces/openshift-network-diagnostics/pods/network-check-target-zzc9f/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.226331882Z namespaces/openshift-network-diagnostics/pods/network-check-target-zzc9f/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.226357412Z namespaces/openshift-network-diagnostics/pods/network-check-target-zzc9f/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.226441354Z namespaces/openshift-network-diagnostics/pods/network-check-target-zzc9f/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.226510315Z namespaces/openshift-network-diagnostics/pods/network-check-target-zzc9f/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.226536676Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.226585857Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.226629048Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.226635558Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.226674299Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.226761591Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.226810112Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.226849473Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.226968405Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.227023276Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.227071678Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.227116258Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.227162279Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.227223651Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.227268712Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.227314072Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.227355524Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.227438835Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.227516777Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.227595949Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.22765155Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.227699041Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.227743912Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.227775253Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.227861114Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.227953506Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.227995667Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.228084179Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.22812801Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.22813816Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.228174361Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.228228072Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.228270743Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.228375775Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.228449467Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.228537489Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.228614491Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.228694882Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.228844465Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.228984988Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.229034029Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.229080261Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.229131302Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.229169392Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.229220364Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.229263665Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.229299315Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.229360436Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.229441118Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.22952316Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.229570161Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.229614842Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.229657763Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.229689643Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.229743415Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.229754115Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.229778525Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.230029911Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.230216475Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.230350808Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.230542082Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.230736856Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.230776117Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.230825408Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.230832228Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.230869959Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.230954851Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.231004862Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.231060863Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.231096174Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.231147295Z namespaces/openshift-network-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.231196036Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.231255337Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.231283818Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.231325758Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.23137639Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.231479902Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.231555414Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.231635125Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.231682676Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.231717477Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.231783368Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.231823639Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.23186752Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.231929722Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.232028074Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.232079015Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.232102605Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.232189837Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.232237858Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.232245518Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.232282209Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.23233323Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.232376191Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.232499404Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.232579545Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.232733019Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.23281201Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.233024505Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.233113047Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.233326952Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.233389333Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.233433434Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.233536006Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.233594457Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.233633328Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.233683969Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.23373021Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.233776701Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.233809462Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.233859313Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.233960335Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.234052867Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.234137579Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.23418173Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.234227201Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.234279292Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.234325463Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.234376404Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.234411355Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.234464586Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.234499177Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.234549428Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.234556348Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.234596519Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.234742522Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.234947866Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.235091269Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.235290863Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.235480348Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.235487238Z namespaces/openshift-network-operator/pods/iptables-alerter-6ktzj/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.235518919Z namespaces/openshift-network-operator/pods/iptables-alerter-6ktzj/iptables-alerter-6ktzj.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.23559771Z namespaces/openshift-network-operator/pods/iptables-alerter-6ktzj/iptables-alerter/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.23560465Z namespaces/openshift-network-operator/pods/iptables-alerter-6ktzj/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.235611801Z namespaces/openshift-network-operator/pods/iptables-alerter-6ktzj/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.235639811Z namespaces/openshift-network-operator/pods/iptables-alerter-6ktzj/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.235729923Z namespaces/openshift-network-operator/pods/iptables-alerter-6ktzj/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.235797794Z namespaces/openshift-network-operator/pods/iptables-alerter-6ktzj/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.235824055Z namespaces/openshift-network-operator/pods/iptables-alerter-l8jfw/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.235861726Z namespaces/openshift-network-operator/pods/iptables-alerter-l8jfw/iptables-alerter-l8jfw.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.235974868Z namespaces/openshift-network-operator/pods/iptables-alerter-l8jfw/iptables-alerter/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.235985138Z namespaces/openshift-network-operator/pods/iptables-alerter-l8jfw/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.235989078Z namespaces/openshift-network-operator/pods/iptables-alerter-l8jfw/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.236027089Z namespaces/openshift-network-operator/pods/iptables-alerter-l8jfw/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.236121771Z namespaces/openshift-network-operator/pods/iptables-alerter-l8jfw/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.236187673Z namespaces/openshift-network-operator/pods/iptables-alerter-l8jfw/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.236227624Z namespaces/openshift-network-operator/pods/iptables-alerter-qnv59/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.236255324Z namespaces/openshift-network-operator/pods/iptables-alerter-qnv59/iptables-alerter-qnv59.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.236328226Z namespaces/openshift-network-operator/pods/iptables-alerter-qnv59/iptables-alerter/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.236335016Z namespaces/openshift-network-operator/pods/iptables-alerter-qnv59/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.236338846Z namespaces/openshift-network-operator/pods/iptables-alerter-qnv59/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.236373027Z namespaces/openshift-network-operator/pods/iptables-alerter-qnv59/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.236461999Z namespaces/openshift-network-operator/pods/iptables-alerter-qnv59/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.23652775Z namespaces/openshift-network-operator/pods/iptables-alerter-qnv59/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.236574101Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.236587541Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.236647703Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.236695854Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.236723554Z namespaces/openshift-node/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.236733745Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.236738435Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.236789556Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.23697644Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.237159174Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.237292257Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.237503901Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.237690335Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.237729156Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.237798097Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.237826788Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.237896819Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.237996121Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.238087754Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.238157215Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.238243707Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.238279178Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.238321428Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.238360089Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.23841093Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.238490092Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.238522173Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.238571234Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.238657856Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.238691107Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.238735977Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.2388388Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.239034114Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.239123236Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.239205078Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.239281019Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.239404142Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.239604746Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.239668437Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.239698318Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.239763419Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.23979704Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.239847071Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.239892362Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.240013085Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.240086986Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.240135447Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.240173378Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.240225009Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.24027325Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.240331652Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.240374943Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.240441454Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.240451684Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.240484205Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.240631968Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.240819392Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.240996956Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.24119556Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.241399725Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.241433385Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.241490376Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.241523877Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.241584058Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.241611649Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.24166912Z namespaces/openshift-operators/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.241680091Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.241685841Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.241709171Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.241858474Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.242101309Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.242251363Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.242443987Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.242626551Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.242656031Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.242716693Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.242755094Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.242820615Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.242854506Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.242930517Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.242983818Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.243167072Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.243242434Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.243321446Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.243368907Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.243401397Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.243591672Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.243620472Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.243675403Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.243715834Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.243797116Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.243839337Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.243900608Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.24400266Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.244045911Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.244052961Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.244091912Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.244142353Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.244185364Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.244390918Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.24447413Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.244782677Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.24492815Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.245465241Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.245544673Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.245702277Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.245756398Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.245815799Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.245931502Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.246058264Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.246103335Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.246190997Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.246238208Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.246279109Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.24633316Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.246374501Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.246437882Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.246478373Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.246522844Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.246529175Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.246569395Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.246620126Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.246649007Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.246735429Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.247477785Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.247527276Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.247566077Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.247622578Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.247676389Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.247750161Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.247786761Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.247842992Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.247851803Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.247898564Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.248012966Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.248054677Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.248099958Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.248145549Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.248152439Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.24819976Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.248343103Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.248529667Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.248732132Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.248963747Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.249155891Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.249163051Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.249229362Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ovnkube-node-ccfz9.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.249414486Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/kube-rbac-proxy-node/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.249422736Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.249426857Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.249462977Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.24959Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.249659422Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.249686862Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.249691592Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.249697502Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.249743893Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.249862696Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250004639Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250036939Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/nbdb/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.25005264Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/nbdb/nbdb/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.25005658Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/nbdb/nbdb/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250091271Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/nbdb/nbdb/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250192273Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250255194Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250297255Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/northd/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250305025Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/northd/northd/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250309235Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/northd/northd/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250342486Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/northd/northd/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250446958Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.25051571Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/northd/northd/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250550601Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ovn-acl-logging/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250557151Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250561171Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250596842Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250685544Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250753045Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250788506Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ovn-controller/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250796256Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ovn-controller/ovn-controller/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250800326Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.250834487Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.251192214Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.251256706Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.251296347Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ovnkube-controller/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.251303877Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.251352098Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.251396599Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.253119586Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.253189817Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.253232278Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/sbdb/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.253243538Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/sbdb/sbdb/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.253249608Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/sbdb/sbdb/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.253276489Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/sbdb/sbdb/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.253389002Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.253462953Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ccfz9/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.253482484Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.253558415Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ovnkube-node-l5mxg.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.253716218Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/kube-rbac-proxy-node/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.253724939Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.253729829Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.253755659Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.253900153Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.253986404Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254025725Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254034655Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254039445Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254063976Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254197709Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.25426817Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254303941Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/nbdb/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254315511Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/nbdb/nbdb/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254320521Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/nbdb/nbdb/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254326902Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/nbdb/nbdb/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254429544Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254499875Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254534796Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/northd/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254550336Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/northd/northd/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254554646Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/northd/northd/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254575537Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/northd/northd/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254684709Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.25475162Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/northd/northd/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254778961Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ovn-acl-logging/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254786721Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254791461Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254833292Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.254946235Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.255033407Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.255064637Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ovn-controller/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.255072618Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ovn-controller/ovn-controller/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.255077127Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.255107018Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.255365204Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.255435255Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.255461626Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ovnkube-controller/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.255468436Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.255474306Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.255513467Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.257147462Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.257217993Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.257245714Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/sbdb/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.257254124Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/sbdb/sbdb/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.257261545Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/sbdb/sbdb/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.257297035Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/sbdb/sbdb/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.257418068Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.257486569Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-l5mxg/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.25750772Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.257586911Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/ovnkube-node-wrg24.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.257747935Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/kube-rbac-proxy-node/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.257761265Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.257766185Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.257787966Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.25797787Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258051351Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258076862Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258084172Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258088992Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258136543Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258257886Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258323767Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258356238Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/nbdb/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258362918Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/nbdb/nbdb/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258366988Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/nbdb/nbdb/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258406259Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/nbdb/nbdb/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258502641Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258576863Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258603403Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/northd/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258609833Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/northd/northd/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258620954Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/northd/northd/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258650654Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/northd/northd/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258755057Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258828688Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/northd/northd/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258860469Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/ovn-acl-logging/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258867509Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.258871619Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.25893937Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.259044273Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.259075063Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/ovn-controller/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.259089694Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/ovn-controller/ovn-controller/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.259096424Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.259130434Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.259446711Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.259516403Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.259542723Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/ovnkube-controller/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.259549414Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.259555263Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.259593784Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.261556917Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.261629828Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.261662339Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/sbdb/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.261670049Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/sbdb/sbdb/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.261674749Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/sbdb/sbdb/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.2617155Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/sbdb/sbdb/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.261811822Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.261896604Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-wrg24/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.262007346Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.262039377Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.262096698Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.262105278Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.262133909Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.262234391Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.262353163Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.262388524Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.262440595Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.262530707Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.262614279Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.26266261Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.262694401Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.262748772Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.262755722Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.262759472Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.262786553Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.262845484Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.262852604Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.262890895Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.2631227Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.263342605Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.263489188Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.263698832Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.263935708Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.263993939Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.26406044Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.264102811Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.264154902Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.264193313Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.264301935Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.264414088Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.264496669Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.264547041Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.264584211Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.264622362Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.264667543Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.264749375Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.264784636Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.264830827Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.264945569Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.26500116Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.265009771Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.265047361Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.265093082Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.265129653Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.265248585Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.265328177Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.265532542Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.265609293Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.265726096Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.265851998Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.266063513Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.266122104Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.266165785Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.266217216Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.266254407Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.266299198Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.266335169Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.266417591Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.266500602Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.266536013Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.266584844Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.266641335Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.266675916Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.266730007Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.266764128Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.266816449Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.266823419Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.26685307Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.267079505Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.267259519Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.267396202Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.267591456Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.26777865Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.26778564Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-lrl7s/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.267819781Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-lrl7s/service-ca-operator-5d6c75f84f-lrl7s.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.267920403Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-lrl7s/service-ca-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.267931293Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-lrl7s/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.267935353Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-lrl7s/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.267982474Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-lrl7s/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.268140418Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d6c75f84f-lrl7s/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.268284791Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.268324782Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.268364512Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.268396353Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.268456014Z namespaces/openshift-service-ca/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.268488225Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.268553056Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.268584607Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.268636558Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.268676339Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.268757911Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.268855263Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.268983626Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.269029417Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.269071458Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.269118588Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.269154429Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.269238441Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.269278732Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.269331363Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.269419515Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.269463886Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.269470586Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.269504827Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.269555208Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.269600439Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.269705041Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.269782073Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.269910236Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.270021418Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.270186831Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.270305354Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.270470368Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.270526469Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.27056949Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.27061204Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.270643921Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.270702082Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.270728553Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.270825415Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.270924117Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.270986019Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.271031089Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.271078561Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.271111711Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.271164302Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.271171363Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.271211663Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.271356947Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.27153201Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.271671923Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.27198657Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.272170694Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.272177874Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-6q7mp/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.272205415Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-6q7mp/service-ca-658d7b7c78-6q7mp.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.272291156Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-6q7mp/service-ca-controller/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.272299807Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-6q7mp/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.272304637Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-6q7mp/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.272322167Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-6q7mp/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.272574973Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-6q7mp/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.272645164Z namespaces/openshift-service-ca/pods/service-ca-658d7b7c78-6q7mp/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.272683755Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.272714105Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.272770357Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.272808868Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.272850448Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.27291432Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.272990622Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.273032972Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.273072713Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.273125574Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.273205826Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.273283168Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.27337975Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.273424681Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.273476252Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.273511043Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.273563854Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.273643525Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.273684317Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.273730407Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.273813029Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.27384858Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.273901301Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.274033164Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.274112916Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.274198208Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.274276599Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.274357731Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.274474423Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.274628847Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.274713219Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.274757029Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.27480444Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.274847381Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.274894992Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.274993274Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.275078036Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.275155218Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.275201499Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.27523655Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.275280171Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.275324792Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.275369462Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.275378203Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.275412423Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.275562087Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.275746881Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.275898484Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.276111148Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.276296202Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.276340373Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.276386994Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.276414435Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.276469686Z namespaces/openshift/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.276515917Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.276564008Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.276611149Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.27665735Z namespaces/openshift/apps/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.276689021Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.276776613Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.276851274Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.276962647Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.277013578Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.277058259Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.277089649Z namespaces/openshift/batch/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.27712801Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.277218972Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.277259713Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.277301144Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.277392266Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.277440677Z namespaces/openshift/core/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.277472927Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.27757528Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.277648951Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.277735113Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.277810485Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.277895607Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.27806231Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.278218063Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.278267335Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.278308925Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.278355946Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.278391587Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.279669125Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.279710706Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.279813438Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.279893259Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.280034242Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.280142725Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.280231537Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.28038909Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.280553274Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.280675816Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.280767458Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.2808567Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.281018323Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.281126396Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.281233338Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.28132883Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.281425832Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.281528714Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.281643557Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.281757729Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.281853381Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.281990074Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.282089237Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.282183259Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.282286601Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.282380003Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.282482235Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.282577447Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.282686469Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.282802692Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.282971156Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.283090358Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.28320404Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.283293072Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.283402325Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.283488647Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.283609879Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.283754862Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.283842474Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.283996167Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.284101029Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.284214192Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.284346645Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.284437397Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.284530449Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.284652672Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.284783854Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.284898357Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.28503695Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.285128302Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.285223434Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.285327406Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.285455379Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.285572661Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.285694754Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.285805896Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.285955629Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.286064202Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.286155054Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.286277776Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.286382678Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.286400779Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.286499921Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.286577713Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.286618314Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.286657434Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.286707645Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.286746696Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.286795767Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.286806098Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.286832348Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.287041173Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.287217906Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.28735851Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.1.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.287560114Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.2.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.287739828Z namespaces/openshift/policy/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.287782068Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.287825749Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.287870141Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.287951962Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.287964303Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.288003353Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.288110426Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.288228218Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.28834086Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.288450513Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.288609646Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.288716448Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.288834761Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.288978604Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.289109277Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.289224349Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.289342862Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.289481655Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.289617818Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.28972023Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.289838783Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.289979536Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.290100808Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.29020516Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.290320143Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.290424715Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.290526987Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.29063138Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.290740812Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.290844594Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.291054639Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.291169041Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.291272503Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.291385026Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.291493558Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.29159451Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.291695372Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.291792265Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.291947098Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.292078711Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.292179623Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.292279595Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.292376757Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.292493089Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.292599162Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.292704684Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.292848307Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.29298224Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.293191404Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.293325357Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.29345527Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.293602663Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.293729536Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.293852409Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.293991461Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.294107374Z network_logs/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.294138295Z network_logs/cluster_scale [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.294215556Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.294294948Z network_logs/multi-networkpolicy [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.29436988Z network_logs/net-attach-def [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.294462082Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.294603285Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.294695637Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.296966845Z nodes/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.296997926Z nodes/debug [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.297089788Z nodes/ip-10-0-138-218.ec2.internal/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.297120569Z nodes/ip-10-0-138-218.ec2.internal/cpu_affinities.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.297545318Z nodes/ip-10-0-138-218.ec2.internal/dmesg [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.297826914Z nodes/ip-10-0-138-218.ec2.internal/ethtool_channels [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.297900415Z nodes/ip-10-0-138-218.ec2.internal/ethtool_features [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.298008038Z nodes/ip-10-0-138-218.ec2.internal/ip-10-0-138-218.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.298432237Z nodes/ip-10-0-138-218.ec2.internal/irq_affinities.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.29858258Z nodes/ip-10-0-138-218.ec2.internal/lscpu [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.298709923Z nodes/ip-10-0-138-218.ec2.internal/lspci [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.298804575Z nodes/ip-10-0-138-218.ec2.internal/podresources.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.298938027Z nodes/ip-10-0-138-218.ec2.internal/pods_info.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.2990694Z nodes/ip-10-0-138-218.ec2.internal/proc_cmdline [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.299153222Z nodes/ip-10-0-138-218.ec2.internal/sysinfo.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.300874759Z nodes/ip-10-0-138-218.ec2.internal/sysinfo.tgz [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.301066913Z nodes/ip-10-0-139-81.ec2.internal/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.301099504Z nodes/ip-10-0-139-81.ec2.internal/cpu_affinities.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.30138226Z nodes/ip-10-0-139-81.ec2.internal/dmesg [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.301683057Z nodes/ip-10-0-139-81.ec2.internal/ethtool_channels [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.301759518Z nodes/ip-10-0-139-81.ec2.internal/ethtool_features [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.30184464Z nodes/ip-10-0-139-81.ec2.internal/ip-10-0-139-81.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.302266339Z nodes/ip-10-0-139-81.ec2.internal/irq_affinities.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.302412072Z nodes/ip-10-0-139-81.ec2.internal/lscpu [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.302535765Z nodes/ip-10-0-139-81.ec2.internal/lspci [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.302622077Z nodes/ip-10-0-139-81.ec2.internal/podresources.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.302740909Z nodes/ip-10-0-139-81.ec2.internal/pods_info.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.302858752Z nodes/ip-10-0-139-81.ec2.internal/proc_cmdline [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.302974394Z nodes/ip-10-0-139-81.ec2.internal/sysinfo.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.30464758Z nodes/ip-10-0-139-81.ec2.internal/sysinfo.tgz [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.304796933Z nodes/ip-10-0-140-37.ec2.internal/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.304827944Z nodes/ip-10-0-140-37.ec2.internal/cpu_affinities.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.305170111Z nodes/ip-10-0-140-37.ec2.internal/dmesg [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.305443157Z nodes/ip-10-0-140-37.ec2.internal/ethtool_channels [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.305519369Z nodes/ip-10-0-140-37.ec2.internal/ethtool_features [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.3056027Z nodes/ip-10-0-140-37.ec2.internal/ip-10-0-140-37.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.306143062Z nodes/ip-10-0-140-37.ec2.internal/irq_affinities.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.306282225Z nodes/ip-10-0-140-37.ec2.internal/lscpu [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.306405938Z nodes/ip-10-0-140-37.ec2.internal/lspci [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.306489209Z nodes/ip-10-0-140-37.ec2.internal/podresources.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.306598642Z nodes/ip-10-0-140-37.ec2.internal/pods_info.json [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.306713694Z nodes/ip-10-0-140-37.ec2.internal/proc_cmdline [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.306794186Z nodes/ip-10-0-140-37.ec2.internal/sysinfo.log [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.308516413Z nodes/ip-10-0-140-37.ec2.internal/sysinfo.tgz [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.308663126Z pod_network_connectivity_check/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.308691957Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.308802539Z static-pods/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.308809959Z static-pods/kube-apiserver/ [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.31538137Z [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.31540176Z sent 72,326 bytes received 8,830,652 bytes 5,935,318.67 bytes/sec [must-gather] [must-gather-ng2z5] OUT 2026-07-28T03:09:43.315410671Z total size is 113,229,072 speedup is 12.72 [must-gather] [must-gather ] OUT 2026-07-28T03:09:43.613452808Z namespace/openshift-must-gather-6q4dh deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 992373db-dd4b-4aad-8975-5163ff304560 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.26" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-r2cj8 [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-r2cj8-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-r2cj8 [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 17310159 Jul 28 03:11 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-r2cj8/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts a3807d6] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-r2cj8/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] 5f5a3e2..a3807d6 ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-07-28T03:11:48.833Z","type":3}]